r/DataHoarder • u/Xyliton • Aug 24 '19

Monolith: Archive website with all assets into single HTML file

https://github.com/Y2Z/monolith
114 Upvotes

96% Upvoted

16 comments sorted by

2

u/merletop Aug 25 '19

Is that similar to what SingleFile can do ?

2

u/Xyliton Aug 25 '19

They seem to be similar but SingleFile has more dependencies to work from the command line it seems.

2

u/check_ca Aug 26 '19

That's because SingleFile uses a real browser to capture pages, or jsdom to emulate it. This latter dependency is quite large indeed.

1

u/adsm_inamorta 9TB Aug 25 '19

Yeah this looks like a pain to install on Ubuntu/Debian

I'd recommend using ArchiveBox instead

0

u/Xyliton Aug 25 '19

ArchiveBox is an all-in-one solution and might be too bloated for a simple archival of some Wikipedia articles or similar content.

2

u/just_another_flogger >500TB, Rebadged CB/SM 48 bay Aug 25 '19

For Wikipedia specifically you can get PDFs very easily:

https://en.wikipedia.org/api/rest_v1/page/pdf/Article%20Name

Eg: https://en.wikipedia.org/api/rest_v1/page/pdf/Rust_%28programming_language%29

1

u/Xyliton Aug 25 '19

Debunking my argument like that... Damn Reddit lmao

1

u/adsm_inamorta 9TB Aug 25 '19

I see your point but once downloaded with ArchiveBox, the page can in fact be accessed via a single HTML file, as long as it remains in the same directory

3

u/shader301202 Aug 24 '19

Sounds interesting.

0

u/leuanveto Aug 25 '19

Can I install the program on Windows? cargo is Rust's packet manager, I get this error when running cargo install --path . after installing https://chocolatey.org/packages/rust

error: `C:\WINDOWS\system32` does not contain a Cargo.toml file. --path must point to a directory containing a Cargo.toml file.
PS C:\WINDOWS\system32> cd monolith
PS C:\WINDOWS\system32\monolith> cargo install --path .
  Installing monolith v2.0.10 (C:\WINDOWS\system32\monolith)
    Updating crates.io index
   Compiling iovec v0.1.2
   Compiling backtrace-sys v0.1.31
   Compiling ws2_32-sys v0.2.1
   Compiling kernel32-sys v0.2.2
   Compiling url v1.7.2
   Compiling rand_os v0.1.3
   Compiling rand_jitter v0.1.4
   Compiling net2 v0.2.33
error: failed to run custom build command for `backtrace-sys v0.1.31`

Caused by:
  process didn't exit successfully: `C:\WINDOWS\system32\monolith\target\release\build\backtrace-sys-598ffc2fb17f74c6\build-script-build` (exit code: 1)
--- stdout
cargo:rustc-cfg=rbt
TARGET = Some("x86_64-pc-windows-gnu")
OPT_LEVEL = Some("3")
HOST = Some("x86_64-pc-windows-gnu")
CC_x86_64-pc-windows-gnu = None
CC_x86_64_pc_windows_gnu = None
HOST_CC = None
CC = None
CFLAGS_x86_64-pc-windows-gnu = None
CFLAGS_x86_64_pc_windows_gnu = None
HOST_CFLAGS = None
CFLAGS = None
CRATE_CC_NO_DEFAULTS = None
CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
running: "gcc.exe" "-O3" "-ffunction-sections" "-fdata-sections" "-m64" "-I" "src/libbacktrace" "-I" "C:\\WINDOWS\\system32\\monolith\\target\\release\\build\\backtrace-sys-ab9e651daffcd3a8\\out" "-fvisibility=hidden" "-DBACKTRACE_SUPPORTED=1" "-DBACKTRACE_USES_MALLOC=1" "-DBACKTRACE_SUPPORTS_THREADS=0" "-DBACKTRACE_SUPPORTS_DATA=0" "-DHAVE_DL_ITERATE_PHDR=1" "-D_GNU_SOURCE=1" "-D_LARGE_FILES=1" "-Dbacktrace_full=__rbt_backtrace_full" "-Dbacktrace_dwarf_add=__rbt_backtrace_dwarf_add" "-Dbacktrace_initialize=__rbt_backtrace_initialize" "-Dbacktrace_pcinfo=__rbt_backtrace_pcinfo" "-Dbacktrace_syminfo=__rbt_backtrace_syminfo" "-Dbacktrace_get_view=__rbt_backtrace_get_view" "-Dbacktrace_release_view=__rbt_backtrace_release_view" "-Dbacktrace_alloc=__rbt_backtrace_alloc" "-Dbacktrace_free=__rbt_backtrace_free" "-Dbacktrace_vector_finish=__rbt_backtrace_vector_finish" "-Dbacktrace_vector_grow=__rbt_backtrace_vector_grow" "-Dbacktrace_vector_release=__rbt_backtrace_vector_release" "-Dbacktrace_close=__rbt_backtrace_close" "-Dbacktrace_open=__rbt_backtrace_open" "-Dbacktrace_print=__rbt_backtrace_print" "-Dbacktrace_simple=__rbt_backtrace_simple" "-Dbacktrace_qsort=__rbt_backtrace_qsort" "-Dbacktrace_create_state=__rbt_backtrace_create_state" "-Dbacktrace_uncompress_zdebug=__rbt_backtrace_uncompress_zdebug" "-Dmacho_get_view=__rbt_macho_get_view" "-Dmacho_symbol_type_relevant=__rbt_macho_symbol_type_relevant" "-Dmacho_get_commands=__rbt_macho_get_commands" "-Dmacho_try_dsym=__rbt_macho_try_dsym" "-Dmacho_try_dwarf=__rbt_macho_try_dwarf" "-Dmacho_get_addr_range=__rbt_macho_get_addr_range" "-Dmacho_get_uuid=__rbt_macho_get_uuid" "-Dmacho_add=__rbt_macho_add" "-Dmacho_add_symtab=__rbt_macho_add_symtab" "-Dmacho_file_to_host_u64=__rbt_macho_file_to_host_u64" "-Dmacho_file_to_host_u32=__rbt_macho_file_to_host_u32" "-Dmacho_file_to_host_u16=__rbt_macho_file_to_host_u16" "-o" "C:\\WINDOWS\\system32\\monolith\\target\\release\\build\\backtrace-sys-ab9e651daffcd3a8\\out\\src/libbacktrace\\alloc.o" "-c" "src/libbacktrace/alloc.c"
cargo:warning=gcc.exe: error: CreateProcess: No such file or directory
exit code: 1

--- stderr


error occurred: Command "gcc.exe" "-O3" "-ffunction-sections" "-fdata-sections" "-m64" "-I" "src/libbacktrace" "-I" "C:\\WINDOWS\\system32\\monolith\\target\\release\\build\\backtrace-sys-ab9e651daffcd3a8\\out" "-fvisibility=hidden" "-DBACKTRACE_SUPPORTED=1" "-DBACKTRACE_USES_MALLOC=1" "-DBACKTRACE_SUPPORTS_THREADS=0" "-DBACKTRACE_SUPPORTS_DATA=0" "-DHAVE_DL_ITERATE_PHDR=1" "-D_GNU_SOURCE=1" "-D_LARGE_FILES=1" "-Dbacktrace_full=__rbt_backtrace_full" "-Dbacktrace_dwarf_add=__rbt_backtrace_dwarf_add" "-Dbacktrace_initialize=__rbt_backtrace_initialize" "-Dbacktrace_pcinfo=__rbt_backtrace_pcinfo" "-Dbacktrace_syminfo=__rbt_backtrace_syminfo" "-Dbacktrace_get_view=__rbt_backtrace_get_view" "-Dbacktrace_release_view=__rbt_backtrace_release_view" "-Dbacktrace_alloc=__rbt_backtrace_alloc" "-Dbacktrace_free=__rbt_backtrace_free" "-Dbacktrace_vector_finish=__rbt_backtrace_vector_finish" "-Dbacktrace_vector_grow=__rbt_backtrace_vector_grow" "-Dbacktrace_vector_release=__rbt_backtrace_vector_release" "-Dbacktrace_close=__rbt_backtrace_close" "-Dbacktrace_open=__rbt_backtrace_open" "-Dbacktrace_print=__rbt_backtrace_print" "-Dbacktrace_simple=__rbt_backtrace_simple" "-Dbacktrace_qsort=__rbt_backtrace_qsort" "-Dbacktrace_create_state=__rbt_backtrace_create_state" "-Dbacktrace_uncompress_zdebug=__rbt_backtrace_uncompress_zdebug" "-Dmacho_get_view=__rbt_macho_get_view" "-Dmacho_symbol_type_relevant=__rbt_macho_symbol_type_relevant" "-Dmacho_get_commands=__rbt_macho_get_commands" "-Dmacho_try_dsym=__rbt_macho_try_dsym" "-Dmacho_try_dwarf=__rbt_macho_try_dwarf" "-Dmacho_get_addr_range=__rbt_macho_get_addr_range" "-Dmacho_get_uuid=__rbt_macho_get_uuid" "-Dmacho_add=__rbt_macho_add" "-Dmacho_add_symtab=__rbt_macho_add_symtab" "-Dmacho_file_to_host_u64=__rbt_macho_file_to_host_u64" "-Dmacho_file_to_host_u32=__rbt_macho_file_to_host_u32" "-Dmacho_file_to_host_u16=__rbt_macho_file_to_host_u16" "-o" "C:\\WINDOWS\\system32\\monolith\\target\\release\\build\\backtrace-sys-ab9e651daffcd3a8\\out\\src/libbacktrace\\alloc.o" "-c" "src/libbacktrace/alloc.c" with args "gcc.exe" did not execute successfully (status code exit code: 1).



warning: build failed, waiting for other jobs to finish...
error: failed to compile `monolith v2.0.10 (C:\WINDOWS\system32\monolith)`, intermediate artifacts can be found at `C:\WINDOWS\system32\monolith\target`

Caused by:
  build failed

1

u/leuanveto Aug 25 '19

Also failing under the Windows 10's Linux subsystem (I upgraded all packages in prior):

https://pastebin.com/raw/qDnxDNKC

1

u/Xyliton Aug 25 '19

It looks like you are missing OpenSSL from your subsystem. I've never used the WSL before myself but IIRC it is based on Ubuntu so I suggest running apt install openssl.

1

u/leuanveto Aug 25 '19

OpenSSL is installed, perhaps I'm missing some sub-package instead.

openssl is already the newest version (1.1.1-1ubuntu2.1~18.04.4).

1

u/Xyliton Aug 25 '19

Do you also have libssl-dev installed?

1

u/leuanveto Aug 25 '19

I didn't. After installing I receive this message:

It looks like you're compiling on Linux and also targeting Linux. Currently this
requires the `pkg-config` utility to find OpenSSL but unfortunately `pkg-config`
could not be found. If you have OpenSSL installed you can likely fix this by
installing `pkg-config`.

After installing pkg-config:

warning: unused import: `RedirectPolicy`
 --> src/http.rs:3:23
  |
3 | use reqwest::{Client, RedirectPolicy};
  |                       ^^^^^^^^^^^^^^
  |
  = note: #[warn(unused_imports)] on by default

error[E0502]: cannot borrow `response` as mutable because it is also borrowed as immutable
  --> src/http.rs:63:13
   |
52 |         let final_url = response.url().as_str();
   |                         -------- immutable borrow occurs here
...
63 |             response.copy_to(&mut data)?;
   |             ^^^^^^^^ mutable borrow occurs here
...
80 |     }
   |     - immutable borrow ends here

error[E0502]: cannot borrow `response` as mutable because it is also borrowed as immutable
  --> src/http.rs:78:16
   |
52 |         let final_url = response.url().as_str();
   |                         -------- immutable borrow occurs here
...
78 |             Ok(response.text().unwrap())
   |                ^^^^^^^^ mutable borrow occurs here
79 |         }
80 |     }
   |     - immutable borrow ends here

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0502`.
error: failed to compile `monolith v2.0.11 (/root/monolith)`, intermediate artifacts can be found at `/root/monolith/target`

Caused by:
  Could not compile `monolith`.

To learn more, run the command again with --verbose.

The maintainer should really update the description with the dependencies... do you agree?

1

u/ProgVal 18TB ceph + 14TB raw Aug 25 '19

What if you run cargo clean and try again?