We now include a `shared_memory.bin` file and `privexsnap/` state snapshot folder which matches our block_log and block_log.index file. They are designed to be used with @someguy123's Non-MIRA Low Memory Mode docker images, to avoid needing a full replay. ==================================================================================================================== Latest State Files - 309G block_log - N/A block_log.lz4 - 385M block_log.index - 3.8G privexsnap/ (folder, containing 33 sub-folders) - 25G shared_memory.bin (17G sparse file, containing ~10G of reclaimable empty bytes) - 9.9G shared_memory.bin.lz4 (use 'lz4 -vd --sparse' when decompressing to avoid wasting space caused by empty bytes) Indexed up to 14 JAN 2021 @ 10:40 AM UTC-0 (approx. block 50,401,167) Indexed using: someguy123/hive:v1.24.4 (b6ffc48a37c1) // Branch: v1.24.4 Plugins: witness condenser_api network_broadcast_api rc_api account_by_key database_api block_api Creation Environment: Ubuntu 18.04 Bionic, 4.15.0-70-generic, 16GB DDR4 ECC, 4 virtual Haswell cores, Docker version 19.03.5, build 633a0ea838 CPU Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat Notes: Tested and verified working on Hive v1.24.4 (someguy123/hive:v1.24.4) ==================================================================================================================== **WARNING ABOUT LZ4 / GZ / BZ2 FILES** For various parts of the Hive blockchain state, including the block_log, shared_memory.bin, and native "state snapshots", we often offer a **compressed version** ending in either `.lz4`, `.gz`, `.bz2`, `.xz` or other common compression formats. Due to the fact that publishing both the original (uncompressed) and a compressed version of various pieces of chain state takes up a lot of extra space, sometimes we may choose to stop offering a compressed version of the file/folder, especially if the compression ratio is poor (how well the file compresses), the compression takes a long time due to the size of the file/folder, or other reasons that we don't believe it's worthwhile to offer a compressed version anymore. Another important thing to be aware of - **compressed versions of files/folders may be outdated compared to the uncompressed version**. For large pieces of chain state such as the block_log - it can take several hours (or even DAYS) to compress the file with a reasonable compression ratio. To ensure we can release updates to the chain state within a reasonable amount of time, we: 1. First, we release the uncompressed files/folders, alongside compressed versions of files/folders which compress within <10 mins 2. For any file/folder we haven't yet made a compressed version available for - we leave the compressed versions from the previous update available 3. We compress larger files such as the block_log behind the scenes inside of a different folder (as to not affect downloads of the existing compressed files), and replace the older compressed versions once the compression is completed. **TL;DR;** Compressed versions of files/folders may be outdated compared to the uncompressed versions, and we sometimes stop offering compressed versions of files/folders if we believe it's not worthwhile to do so. ==================================================================================================================== **If you are using Hive-in-a-box (hive-docker), rather than trying to download the files yourself, use the built-in command** You can use `fix-blocks` to interactively download each part of the chain state from Privex's file server, including the `block_log`, `block_log.index`, and `privexsnap/` state snapshot folder. The `fix-blocks` command will prompt you before each component download, asking whether you need/want to download or update your local copy of that component from our file servers. Interactive `fix-blocks` - download everything, or just individual components, by answering some yes/no prompts: ./run.sh fix-blocks To *automatically* download just the `block_log`: ./run.sh fix-blocks blocklog To *automatically* download just the `block_log.index`: ./run.sh fix-blocks index To *automatically* download just the `privexsnap/` folder: ./run.sh fix-blocks snapshot ==================================================================================================================== To download a the state snapshot files, we recommend using rsync. Download the block_log.lz4 using `wget` with lz4 on-the-fly decompression, and the block_log.index (no compression), then use rsync to download the rocksdb folder. # If you're downloading the block_log from FRESH - i.e. you don't have an existing one wget -q http://files.privex.io/hive/block_log.lz4 -O - | lz4 -d - /root/hive-docker/data/witness_node_data_dir/blockchain/block_log # If you have an existing intact block_log, you can update it using Rsync - downloading only the remaining part of the block_log # that you're missing from your local copy, rather than re-downloading it from scratch, by using rsync --append rsync -avh --append --inplace rsync://files.privex.io/hive/block_log /root/hive-docker/data/witness_node_data_dir/blockchain/block_log # Download the block_log.index via wget (or rsync - doesn't matter much which method you choose) wget http://files.privex.io/hive/block_log.index -O /root/hive-docker/data/witness_node_data_dir/blockchain/block_log.index # Download the Hive "state snapshot" folder using Rsync rsync -av --progress --delete rsync://files.privex.io/hive/privexsnap/ /root/hive-docker/data/witness_node_data_dir/snapshot/privexsnap/ ====================================================================================================================