Bacalhau Project Report - Nov 11, 2022
Bacalhau goes beta! With WASM, FIL+ and DAG previews!
Hi everyone! Luke here, tech lead on the Bacalhau project. This is the first time I’ve posted a project report on the blog, the previous ones can all be found linked to from here.
Engineering update
We just got back from a brilliant, energizing week in Lisbon at the CoD (compute over data) summit. Here are the youtube links:
We did loads of work in the run-up to the summit to get the network ready for prime-time, and we’re happy to announce that Bacalhau is now in Beta!
Also check out our talks:
Bacalhau - a Platform to Bring Compute to Big Data: A summary of the progress of Bacalhau towards Beta, and worth watching for probably the sickest demo ever that involves Stable Diffusion and cheese!
Bacalhau Deep Dive: The Path Ahead: talking about the roadmap and challenges we see ahead
FIL+ and Bacalhau: On how we might find a “bridge” to incentivize storage providers to become computer providers in the near term (before Bacalhau has its own incentive layer) - including a demo of our new dashboard!
Determinism in Bacalhau with WASM: Showing our latest work on WASM and a deterministic WASI implementation that can be used to run arbitrary Rust (& more) code - with a great demo of running WASM jobs that do image manipulation.
Pipelines in Bacalhau: showing our latest work on Airflow integration with Bacalhau, so you can use Airflow to orchestrate a graph of Bacalhau jobs, feeding the output of some into the input of others
It was great to meet loads of people with similar interests including lots of other folks working on CoD projects and folks with experience of GridCoin and the BOINC project!
Dashboard and FIL+
We now have a beautiful web dashboard showing the status of jobs on the network!
This will be deployed soon, and additionally will allow notaries to log in in order to mark certain jobs as eligible for FIL+ rewards so that service providers get first bids on the storage of the FIL+ data which can be 10-35x more valuable to store on the Filecoin network.
WASM updates
It’s now possible to run WASM jobs with environment variables, with programs specified as CIDs, and inputs and outputs as files! Here’s an example of a WASM job that runs seam a carving algorithm on input data that was generated by Stable Diffusion on GPUs in Docker!
Pipelines in Bacalhau
You can now wire Bacalhau jobs together using the experimental Airflow integration! Here’s an example of a Bacalhau pipeline in Airflow using the new BacalhauWasmRunJobOperator
and BacalhauDockerRunJobOperator
🎉
Network stability
We’ve done a lot of work on improving the network stability, including switching from ad-hoc connections to using Direct Peers in libp2p so that the system will reconnect on network failures. The network worked for all the live demos in the talks at the conference, and is still going strong 🎉
What’s next
We’re actively working on making the tests more robust and faster
We’re running experiments to scale the network further at global latencies
We’re continuing work on the simulator, and digging in more to game theory
For more detail on the planned work ahead, watch Bacalhau Deep Dive: The Path Ahead.
Thank you very much!