Because of the ongoing performance issues, I moved my whole Friendica database of opensocial.at and friendica.me to a dedicated root-server just for mariadb with 128GB RAM.
At first, it seemed like it helped and opensocial.at is faster than before (the performance of friendica.me is still a mess, but I will analyze it later), but the /photo/preview route is now going crazy:
It seems like the RAM I/O issue I had (which is now gone) is now replaced by an Network throughput issue - but at least the network I/O seems fine:
Do you have any idea what I can do now?
One thing would be to add a "StorageClass" for the Photo to load them (directly) from a filedirectory/block storage instead of the database.
@Philipp Holzer .. I see Fatal Error (E_ERROR): gd-webp cannot allocate temporary buffer all the time.. could this be related to the "/photo/preview route" ?
Ich starte nun eine Migration von opensocial.at auf eine stärkere server-hardware. Dadurch entstehen temporäre Downtimes in den kommenden Stunden.
Ich möchte mich für die Unannehmlichkeiten entschuldigen, aber ich denke, dass dadurch die gesamte Performance von opensocial.at wieder besser werden sollte :).
Lg, Philipp
----
Dear opensocial.at user,
I'm planning to migrate my server infrastructure onto a more powerful environment. The goal is to increase the performance, as, I think, you all currently experience a downgrading performance.
I hope it will help :)
It will take some hours, where opensocial.at will be temporary unavailable from time to time.
I'm planning to migrate my server infrastructure onto a more powerful environment. The goal is to increase the performance, as, I think, you all currently experience a downgrading performance.
I hope it will help :)
It will take some hours, where friendica.me will be temporary unavailable from time to time.
@Philipp Holzer Temporary install/enable xdebug and set mode=trace. Then use e.g. kcachegrind to analyze it, e.g. which method takes the most time. That might be a start! Yes, you normally want to have xdebug disabled on live (no development) servers. But this is an exception as high load cannot be simulated properly to find the bottleneck.
You can enable the asynchronous with 'decoupled_receiver' => true, in your config. Then the inbox requests would be much faster. Then the processing is done via worker processes.
But that one is tricky to set up. Best is to store the photos in a folder outside the web folder and then you can define a rule in your webserver that will redirect requests to the avatar folder to that folder.
@Michael Vogel @Philipp Holzer Will these cached avatars be checked if they are outdated? I currently have trouble getting @Tobias 's avatar over here.
It will become extremely large over time. So storing in some separate folder might be better for backup purposes or for keeping your web root clean. Also there is an option for an alternate URL. So you could even store these files on some separate web server that only serves static content.
No, the setup process is completely different. You have to have a path (somewhere) that is both readable and writable by the frontend and backend process This path needs to be reachable when you call https://your.server.tld/avatar (you have to redirect /avatar) or you have to define avatar_cache_url.
You you setup the file path from above in avatar_cache_path and set avatar_cache to true. After you deactivate the caching of the avatars in the admin frontend, it should work.
The URL path is stored in the contact table fields. So once you defined that URL, you mustn't change it again, since all old requests would fail. This mechanism is powerful, but currently too easy to misconfigure, that's why it is only accessible via the config file.
@Michael Vogel @Philipp Holzer U oh, writable and readable by the frontend, means web server? Do you mean here also through https://? That's never a good idea.
This image is now served by a completely different nginx than the frontend nginx . Because it's stateless, I'm now able to start more nginx for "just" serving stateless static content!
Old avatars still will be served the old way and will be replaced, once they are updated - which can take weeks. But I guess I wrote some script that speeds up this process.
Hi, after moving all contact images to avatar/, there are still a lot of long running requests, see the image above. Do you have any idea? The overall CPU/RAM is on a normal state.
I'm currently reducing the whole BaseUrl.php code massively. Do we really need the ssl_policy, urlpath and hostname separate from the system.url?
I will use for the BaseUrl.php, based on the system.url a "real" UriInterface as $this->url, so we don't need saving the scheme, urlpath and hostname separately anymore. They are just useful for the install process but must not be changed afterwards.
The only thing, which I'm unsure is the ssl_policy, because if someone changes it afterwards in the admin site, all URL in all contacts and photos will get updated. But the question is => is this even allowed? I think this could brick the access over federation because the base-url of each entry isn't right anymore. And it isn't supported when the policy is changed by console.
So I would drop it as well and merge all config entries into the system.url.
Additionally, I will replace the Exception with a "CRITICAL" log entry to avoid a WSOD.
@Philipp Holzer This sounds good, internally we don't make a difference between HTTP and HTTPS URLs for remote servers (thanks to the nurl) so it wouldn't have an impact on Friendica federation.
The SSL-Policy was a thing of ancient times where you could define if you had a self certified certificate, a real one or none. There had been some code around this. Just have a look if this is used somewhere.
@Tobias I'm not sure what wouldn't work with a self-signed certificate. Unless we are checking the CA for self-requests, but I'm not sure the legacy SSL Policy configuration value is about that.
@Philipp Holzer For a local installation, like https://friendica.local you definitely need to ignore SSL/certificate errors because they are mostly only self-signed.
It seems like I do have a problem with my opensocial.at database .
The database ran full about a year ago (what a shame, I know :( ...) and I had to recover some of the data (otherwise I had to import a backup about ~12hours ago, I tried to avoid to loose posts for this period of time for my users ...). So I stopped it, started it with innodb_force_recovery=3, repaired it and checked it with mysqlcheck.
Two days ago, I updated the mariadb from 1.10.3 to 1.10.10 and now the problems occur again.
I stopped the MariaDB 1.10.3 container, updated it to 1.10.10 and started it again .. And MariaDB started a "crash recovery". I really don't know why... The crash recovery wasn't successfully (I tried it ~4 times), so I had to add innodb_force_recovery=3 again.
Now the database went up, but everytime I stopped it and started it again, the crash recovery appears again.
So after the instance was up and running, I dumped the whole database with mysqldump into one single *.sql , started a brand new MariaDB 1.10.10 and imported the dump again.
So far so good ..
But... unfortunately, after a restart, the crash recovery appears again. So I'm totally lost, what's now happening..
I noticed during the import that one batch of rows took about 2 hours to complete with the result query affected 0 rows.
here's my customized.cnf, which I'm using (it's a Hetzner root server with 64 GB RAM and 12 CPUs):
Hi Philipp, im not familiar with the Container Versions of MariaDB, because that version numbers don´t fit the official "self installed" ones. Mariadb >10.4 has some weird changes in indexing, which makes the database insanely slow.
You could try to include this into your mariadb.conf:
@Philipp Holzer My guess regarding the permament crash recovery: When sending the database a shutdown, it flushes erverythin in memory in its files. Depending on your hardware and database size, this could take a while - i´ve seen databases that took up to 10 minutes to do that. If your container or OS is shutting down before the process ends, youll get a crash recovery on the next startup. check if the maridb process is terminated before the container or OS is shut down.
If the databse is correctly shut down and the crash recovery starts again on the next database startup, you could try to remove the MariaDB Log files. The transactions are stored in that files. You should back them up before removing them, just in case you need a rollback.
i don´t know your mariadb install locations, but thats what i would do:
Yep, it’s a OOM sigkill from docker. I’m sorry, I don’t use docker and I am not familiar with it - but that Exit Code looks like a good hint to start with :)
and as far as I can say, it would be better if the second route should have an own root, like `dfrn_item/{uri-id}[/conversation.atom`, as it's a DFRN only logic. Am I right so far?
If so, can I move it to this new path or is this path necessary for the federation usage, means it could be called from other servers too? I just found calls with prefix DI::baseUrl(), so I guess it's "just" for the server itself, but I'm not sure ...
I was mixing two and three because the .atom postfix of the item-uri is possible for the third route as well, but you're right, it doesn't seems plausible
ah damn .. my fault .. ooohhh maann thanks for your help and discussion here .. I forgot about the legacy *_content route and thought the only place, where it's called is at update_display
I still struggle at mod/update_display.php It's the only place, where the display_content from mod/display.php is called with the parameter $update = true.
But I don't find any place inside Friendica, where we call /update_display?p=<profile_uid>.
Do you know if this is till in use and if so, what's the meaning of it? If we wouldn't use it anymore, there's a lot of code at mod/display.php, which I could delete in that case..
Currently, we strictly separate between API calls and the Frontend. But, Why?
I'm currently moving the events to the Module directory. And there's already an API for it, but as far as I understand @Michael Vogel , we must not mix API and Frontend.
But as far as I know, in modern environments, there's a backend part (= API), and a frontend part (= view) The own frontend still uses the same API. So we don't have to implement the same business logic at different places. As I said, I came across at Event, where this issue produces significant double-code.
So at least what I would like (in a final state :
API ... Create, Update, Delete, Get, GetMultiple for our Data model, or "everything without HTML output"
API is a misnomer in our case because it can designate several things: - Our internal API (what you described in your post) - The Twitter API endpoints we support - The Mastodon API endpoints we support - The GNU Social API endpoints we support - The Friendica-specific API endpoints we expose (mainly for Friendiqa)
We definitely lack a unified internal API but that’s supposed to be taken care of by the Repository/Factory/Model/Entity structure that we should keep expanding.
@Hypolite Petovan why don't separate the api for Twitter, Mastodon, Gnu etc. and of course an api to communicate with the frontend. No mixing the api together. Activateing the maybe Mastodon APi as main class with subclasses in one folder and subfolders.
So far we’ve done a good job at having the business code identical for the Twitter and Mastodon API endpoints. There are some legacy Friendica endpoints (mostly the asynchronous endpoints used by Javascript) that are redundant with the page modules.
@Hypolite Petovan Can this endpoint not be closed and moved to an other area. So that you have the main friendica core who will do the internal magic. On the other hand the API section who is communicating with the core. Additional an internal API who is communicating with the core and the frontend section. On all APIs there is only data communication.
The Frontend Section is an additional area who will communicate with the frontend API and the user. So the magic on the frontend section will be created only in the frontend and the API communication will only send data ... No html, no Jscript and no other magic.
In that case all three sections (Core, API, Frontend) could be developed separate with less interference between them.
Sure if you need a new function... First core the API and if needed then frontend @Philipp Holzer @Michael Vogel
I don't think we need to go that far. Separating the web frontend from the API would be a massive undertaking at this point, and also there's a trend where even Javascript frontends are starting to request already formed HTML from the backend because it turns out it's faster this way. Which means we're both behind and ahead of the curve!
However we do need to extract the business logic in a unique place so that we can reference it from the places it is duplicated.
@Hypolite Petovan I think this is very necessary. Passing preformatted web content makes you very inflexible since you have to go directly to the core. That's where I see the problem, that you either have a good coder for the core but not a good designer. The other way round a designer of the frontend can't write a good core code. For this reason the Smarty framework is currently running in Friendica, which as I understand the framework wants to separate this.
If you want to speed up the whole thing, you have to get away from PHP code which is executed at runtime and create static pages. Peppered with lots of javascript which offloads/executes the execution to the client.
Thankfully, we can have both HTML modules and API endpoints! As long as we don't duplicate business code, which is what Philipp has been encountering.
On the other hand, writing a full Javascript web frontend is a massive endeavor, there currently are almost 500 web frontend routes that each represent a part of a Friendica feature. They all would need to have a Javascript counterpart, without even counting the corresponding bespoke API endpoints that would need to be created.
@Michael Vogel What is the showstopper for that? Cause that is also a thing for a new Webdesign. So The Web sould be separate from the code with all the magic. @Philipp Holzer
Metadata about each add-on should be in an extra file, I would prefer something like addon.json
I would like to replace the whole Hook class with a proper Registration class, which is loaded when activating the Add-on. So we can remove the whole Hook table.
All Events in Friendica should be in an events.config.php with a proper description (maybe as constants?)
for legacy purpose, I would create a wrapper class like the LegacyModule class, which would execute the corresponding php-functions
I'm thinking about a giving each Event a specific interface (not knowing how tbh currently, maybe the corresponding interface setting as a value for the Event inside the events.config.php?), so each event and each add-on will know exactly what parameters and which result is necessary
This is it so far..
I would like to start with creating the events.config.php and replace all text with constants.. I think this is a non invasive behaviour, but gives some insights about the next steps
@Hypolite Petovan is right, I should have started this conversation first, but it made fun to restructure our code, so it is what it is *gg*
What's the issue/my pain:
I'm thinking about introducing the PSR-7 Request/Response interfaces and create an Emitter, so there's no way out anymore directly inside the code, everything is done by the emitter. With this pattern, we could add all the authentication stuff inside a "Middleware" and get rid of the different auth-places (App, Security\Authentication, Module\BaseApi)
So, now I tried to think where to start. We would need a Middleware directory for all the auth/base-load things, but we already do have a mix of domain-based directories (TowFactorAuth), functionallity based directories (Model/Object/Factory/Collection/Capabilities) and and core directories (Core, App, Console, Worker)...
And I try to start to "really" encapsulate the domain-based directories inside Friendica\Library, so we get rid of some mixings, having all domain based classes inside one directory.
In the end, I think we should have some core directories at src, directing everyone to the right place.
I hope I didn't loose you at least, I just wrote down my path to this PR :)
PS: there're "just" ~200 code changes, the rest is because of the messages.po recreation don't worry :)
I always struggle with our codebase about where to find what content .. Because we moved from pattern to pattern, we left a lot of leftovers in our codebase.
This PR should restructure our codebase...
Thank you for the write-up. I am not familiar with the Emitter pattern, would you mind explaining it more?
Secondly, what is the ultimate structure of the src folder? Is the Library folder transitional or final? In the latter case, what other folders would be at the same level?
The Emitter isn't really a pattern, it's the "final" class, where we pass the whole Response of our Modules or Exits and which prints header, body and HTTP Codes. It's the ending of my start with the `Response` class and it should moveout some of the Page code.
And yes, I think the Friendica\Library is the final place for the domain classes. I think about the following folder structure (which btw. maybe help @Michael Vogel reducing his headache with the structure):
App ... All base-classes, which are necessary for the Request/Response behavior of Friendica
Console ... All Console classes (= the CLI)
Core ... All Core Friendica classes, which are always necessary
Database ... The core Database Namespace (maybe inside Friendica\Core at least ?)
Library ... The whole business logic, domains, Content, (refactored) Models, ...
Module ... The "Glue" between Routes and the business logic (= Library) ... as it is currently :)
Protocol ... Federation specific logic
Worker ... All Worker classes (= the cron)
I believe the rest is "legacy" and should be moved inside one of the namespaces above: Content, Model, Network, Object, Security, Util
And now tbh I'm a little bit struggling where to put a Middleware folder *gg*.
Addressing @Michael Vogel "more complex" .. At least I hope it's getting better with the whole structuring and introducing of Middleware, bringing more focus onto our code.. Still, I believe your main concern is this Factory and Repository and Entity thingy, which will persist
But for example the Mastodon API calls are now bundled inside Friendica\Library\Api instead of floating around at Friendica\Factory, Friendica\Object and Friendica\Collection
Good point .. but at least merging the current namespaces at src into only a few will help us with little effort. The Middleware thingy is just another playground, yes I must confess. I fell into it because I tried to make BasicAuth and OAuth dynamic, to make some of the skipped tests testable ..
sounds logic, but of course thank you for the hint!
Currently, for me it looks like a giant hill to make them testable, that's why I started to refactor them step by step (and therefore first step is to make a base-structure for it )
But you're right, the Addons are really old and when I added the S3 Addon, I saw that there's no possibility to automatically test anything of it because of the current structure ... Maybe that's a better place to focus
I'm with Michael here, before we start tackling the testability of modules (which I am interested in), I would like them to be all moved from the mod/ folder. This doesn't interfere with your proposed restructuration where they wouldn't be moved again.
I'm sorry, I bricked the CI with the latest Woodpecker change.. Now instead that we don't create Archives at all nodes, we now don't execute some pipelines at all and they are "stuck forever" and blocking new executions.
I'm not sure if I can fix it today, so here's the announcement
I will kick the pipelines from time to time to make PR checks possible, but the "merge" pipelines aren't working (so no new archive appears currently..)
It was a mix of wrong environment variable setup for $WOODPECKER_LABELS (added additional " for the value, which got escaped and added to the value) and using the `label` filter in Woodpecker, not knowing that this feature is only in the nightly builds yet
--> as far as I can see, all Archives are now deployed at https://files.friendi.ca as expected :)
Hi, I'd like to add a new domain `docs.friendi.ca` where we store the whole documentation. It should be versioned, like https://docs.friendi.ca/2022.09/ for the docs of the stable 2022.09 version or https://docs.friendi.ca/develop/ for the docs of the develop branch.
I think it would be possible to automatically generate it.
But before, I do need the domain - pinging @Michael Vogel - and a routing to the current files.friendi.ca vm-instance - pinging @utzer ~Friendica~
I'd like to generate it automatically during the CI, so the update would be smooth and automatically
@Philipp Holzer ich habe alles eingerichtet und zum testen liefert docs.friendi.ca jetzt files.friendi.ca aus, ich bin nicht sicher was du genau vor hast, lass mich wissen wenn du was brauchst oder Hilfe benötigst.
I don't have currently that time, so it's still WIP .. When it's finished, of course there will be a sub-dir for stable releases and develop - and Release candidates, similar to our branch structure :)
Hi all, I lost my mobile phone during the last festival and so my TOTP-App was gone forever. Unfortunately, I wasn't aware of my recovery codes, so I thought I would have to reset my accounts of Friendica.
But no! just use select code from 2fa_recorvery_codes where uid = 66 and used is NULL; and voilá, I used the first code and was back in.
@Hypolite Petovan , isn't this a possible security issue, is it?? Storing such sensible data as plaintext. I think we should save it as hash like for passwords to make it impossible to read it again .. Yes, the downside is that there's no possibility to save recovery_codes from the settings-panel again, but tbh I feel a little bit unsafe, but maybe it's just a feeling
We hash passwords not just because we want to prevent someone having a database dump from logging in the website depending on this database, but also for the other websites users might use the same password for. There is no such risk for randomly generated recovery codes.
Additionally, the use of the recovery codes necessitates the use of the password (that you still had in your head/password manager), after all it's a second-factor authentication, which means that it can be freely compromised as long as the first factor (the password) is safe.
Still, we can hash these codes and as a result only show them once, it isn't that hard other than requiring someone™ to spend some quality time on this task.
last week, I struggled with the size of my database for opensocial.at .. It went full (80GB) .. I added additional 10GB to the volume, but it seems to constantly increase and I think in a few weeks, I will be at the same situation like the last week.
Is there a possibility to shrink the database, or to wipe old data? I recently deleted about 1.200 spam accounts (and set the register option to approval first ... lessons learned...). Are there leftovers inside the db which I can delete?
It's not that I don't have enough space, but I'm afraid of my backup borg-backup space in the near future ^^
17000 from 59000 finished ;)
.
xy..
in reply to Philipp Holzer • • •Fatal Error (E_ERROR): gd-webp cannot allocate temporary bufferall the time.. could this be related to the "/photo/preview route" ?Philipp Holzer
in reply to xy.. • • •xy..
in reply to Philipp Holzer • • •/var/www/localhost/friendica/src/Object/Image.phpline 171Michael Vogel
in reply to Philipp Holzer • • •Philipp Holzer
in reply to Michael Vogel • • •Roland Häder
in reply to Philipp Holzer • • •/tmpfolder? Here on my server it ran full withmagick-XXXXXfiles.Philipp Holzer
in reply to Roland Häder • • •But there are just two zero-byte magick files at
/tmp:)