Weekly Dev Update #153

01 June 2021 —Kee Jefferys

Hey y’all,

This week the Session team put a heavy focus on Session Android performance, particularly on issues that affect Session when it is opened and needs to load many conversations at once. The Lokinet team focused on a path handover issue which is causing connection instability. The Oxen core team finished a big update to the Oxen storage server which should speed up the process of fetching messages from the storage server.

Oxen

Oxen Core

Batching for Service Node rewards https://github.com/darcys22/loki-core/commits/batch-sn-payments

Oxen Storage server

Replace boost beast with uWebsockets https://github.com/oxen-io/oxen-storage-server/pull/432

- Remove pre-HF18 legacy code/endpoints that isn't being called anymore

- Remove process_lns_request endpoint (it is broken, and Session clients have already switched to using oxend_rpc with ons_resolve instead).

- Replace boost beast http(s) client code with cpr (https://github.com/whoshuu/cpr); this is a whole lot nicer to use for HTTP requests.

- Replace boost beast http server code with uWebSockets (https://github.com/uNetworking/uWebSockets). This gives a much nicer interface, and makes it easy for us to add websocket support for clients in the future.

- Remove boost::asio; it's not needed anymore with the removal of the above.

- Replace bootstrap RPC code with authenticated, encrypted OMQ RPC.

- Remove boost circular buffer use; a regular map with a two-line trim is simpler for the block hash cache and a limit on stored snodes doesn't seem necessary for the rate limiter.

- Make rate_limiter clean itself periodically rather than keeping buckets around indefinitely

- Make rate_limiter thread-safe so that you don't need to hold the entire service_node_ lock to use it.

- Remove ip_utils; we don't allow redirects and are sufficiently restrictive on the URL target that it seems unnecessary (plus not having it lets us offload DNS lookup to curl as part of the request).

- Replace /swarms/ping_test/v1 with /ping_test/v1; this new request now returns the remote pubkey in a header, and no longer includes an SSL cert signature (so that we can drop the SSL cert signatures after HF19). The old one will still be used until HF19.

- Add OMQ endpoint for storage tests; starting at HF19 it will get used rather than the HTTPS one.

- Refactor storage test retries into request_handler (rather than being in the HTTPS specific code)

- Move HTTPS server-specific code for validating snode signatures from headers out of generic request_handler code and into HTTPS-specific code.

- Make onion proxy-to-url timeout a bit less than the onion request timeout so that the client has a better chance of getting a relayed timeout error (rather than getting a timeout from the edge node).

- Miscellaneous cleanups.

- Logger: use file and line number instead of func because the latter is nearly useless when called from a lambda.

- Shorten timeout values for ping tests, storage tests, and bootstrap connections to 5s, 15s, and 10s, respectively, from.

- Refactor recent stats reporting to use rolling averages that always have 60-70min of stats and drop off 10min at a time, rather than the 1-period hard reset. Also fixes various stats that weren't calculated/reported properly.

- Add onion and proxy requests to systemd status line as well as used database size.

- Enable WAL for sqlite3 database

- Removed buffered message relaying for swarm propagation; it's counterproductive with omq's persistent connections.

- Enable jemalloc by default.

Oxen Mobile Wallet

Fix crash occurring when getting users balance https://github.com/oxen-io/oxen-mobile-wallet/pull/9

Lokinet

This week was spent tracking down Lokinet connectivity issues. We’ve narrowed the problem down to a longstanding issue in path handover (i.e. when one path expires and Lokinet builds and switches to a new path, which is meant to happen seamlessly every few minutes). 0.9.x included various improvements to better detect failing paths, but these appear to have exposed a lurking bug that has been present for quite some time; fixing this to ensure a smoother Lokinet experience is currently our top priority.

Ongoing path handover new diagnostic tools, potential fixes, and various related improvements (draft PR) https://github.com/oxen-io/loki-network/pull/1653
Make DNS work better when no upstream servers are configured https://github.com/oxen-io/loki-network/pull/1656