Weekly Dev Update #153
This week the Session team put a heavy focus on Session Android performance, particularly on issues that affect Session when it is opened and needs to load many conversations at once. The Lokinet team focused on a path handover issue which is causing connection instability. The Oxen core team finished a big update to the Oxen storage server which should speed up the process of fetching messages from the storage server.
Batching for Service Node rewards https://github.com/darcys22/loki-core/commits/batch-sn-payments
Oxen Storage server
Replace boost beast with uWebsockets https://github.com/oxen-io/oxen-storage-server/pull/432
- Remove pre-HF18 legacy code/endpoints that isn't being called anymore
- Remove process_lns_request endpoint (it is broken, and Session clients have already switched to using oxend_rpc with ons_resolve instead).
- Replace boost beast http(s) client code with cpr ( https://github.com/whoshuu/cpr); this is a whole lot nicer to use for HTTP requests.
- Replace boost beast http server code with uWebSockets ( https://github.com/uNetworking/uWebSockets). This gives a much nicer interface, and makes it easy for us to add websocket support for clients in the future.
- Remove boost::asio; it's not needed anymore with the removal of the above.
- Replace bootstrap RPC code with authenticated, encrypted OMQ RPC.
- Remove boost circular buffer use; a regular map with a two-line trim is simpler for the block hash cache and a limit on stored snodes doesn't seem necessary for the rate limiter.
- Make rate_limiter clean itself periodically rather than keeping buckets around indefinitely
- Make rate_limiter thread-safe so that you don't need to hold the entire service_node_ lock to use it.
- Remove ip_utils; we don't allow redirects and are sufficiently restrictive on the URL target that it seems unnecessary (plus not having it lets us offload DNS lookup to curl as part of the request).
- Replace /swarms/ping_test/v1 with /ping_test/v1; this new request now returns the remote pubkey in a header, and no longer includes an SSL cert signature (so that we can drop the SSL cert signatures after HF19). The old one will still be used until HF19.
- Add OMQ endpoint for storage tests; starting at HF19 it will get used rather than the HTTPS one.
- Refactor storage test retries into request_handler (rather than being in the HTTPS specific code)
- Move HTTPS server-specific code for validating snode signatures from headers out of generic request_handler code and into HTTPS-specific code.
- Make onion proxy-to-url timeout a bit less than the onion request timeout so that the client has a better chance of getting a relayed timeout error (rather than getting a timeout from the edge node).
- Miscellaneous cleanups.
- Logger: use file and line number instead of func because the latter is nearly useless when called from a lambda.
- Shorten timeout values for ping tests, storage tests, and bootstrap connections to 5s, 15s, and 10s, respectively, from.
- Refactor recent stats reporting to use rolling averages that always have 60-70min of stats and drop off 10min at a time, rather than the 1-period hard reset. Also fixes various stats that weren't calculated/reported properly.
- Add onion and proxy requests to systemd status line as well as used database size.
- Enable WAL for sqlite3 database
- Removed buffered message relaying for swarm propagation; it's counterproductive with omq's persistent connections.
- Enable jemalloc by default.
Oxen Mobile Wallet
Fix crash occurring when getting users balance https://github.com/oxen-io/oxen-mobile-wallet/pull/9
This week was spent tracking down Lokinet connectivity issues. We’ve narrowed the problem down to a longstanding issue in path handover (i.e. when one path expires and Lokinet builds and switches to a new path, which is meant to happen seamlessly every few minutes). 0.9.x included various improvements to better detect failing paths, but these appear to have exposed a lurking bug that has been present for quite some time; fixing this to ensure a smoother Lokinet experience is currently our top priority.
Ongoing path handover new diagnostic tools, potential fixes, and various related improvements (draft PR) https://github.com/oxen-io/loki-network/pull/1653
Make DNS work better when no upstream servers are configured https://github.com/oxen-io/loki-network/pull/1656
Fix message deletion bug https://github.com/oxen-io/session-android/pull/593 , https://github.com/oxen-io/session-android/pull/573
Update Hindi translation, (thanks to community member iArchitSharma) https://github.com/oxen-io/session-android/pull/586
Update readme https://github.com/oxen-io/session-android/pull/582
Add resolution of ONS names https://github.com/oxen-io/session-android/pull/574
Batch conversations updates https://github.com/oxen-io/session-android/pull/570
More efficient thread trimming and database clean-up https://github.com/oxen-io/session-android/pull/568
Add saved attachment notifications https://github.com/oxen-io/session-desktop/pull/1672
Deduplicate messages based on server timestamp https://github.com/oxen-io/session-desktop/pull/1670
Visual display of current onion request path https://github.com/oxen-io/session-desktop/pull/1665
Remove V1 file server and V1 open group code https://github.com/oxen-io/session-desktop/pull/1671
Do not overwrite attachment name https://github.com/oxen-io/session-desktop/pull/1667
Remove unstable file server dialogue https://github.com/oxen-io/session-desktop/pull/1666
iOS deep link for Session IDs / Usernames (thanks to community contributor iArchitSharma) https://github.com/oxen-io/session-ios/pull/421
Full Hindi translation (thanks to community contributor iArchitSharma) https://github.com/oxen-io/session-ios/pull/420
Performance improvements https://github.com/oxen-io/session-ios/pull/419