diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f658dbf81..6edfea44c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,22 +1,12 @@ - + # Contributing to coreutils -Contributions are very welcome, and should target Rust's main branch until the -standard libraries are stabilized. You may *claim* an item on the to-do list by -following these steps: - -1. Open an issue named "Implement [the utility of your choice]", e.g. "Implement - ls". -1. State that you are working on this utility. -1. Develop the utility. -1. Add integration tests. -1. Add the reference to your utility into Cargo.toml and Makefile. -1. Remove utility from the to-do list in the README. -1. Submit a pull request and close the issue. - -The steps above imply that, before starting to work on a utility, you should -search the issues to make sure no one else is working on it. +Contributions are very welcome via Pull Requests. If you don't know where to +start, take a look at the +[`good-first-issues`](https://github.com/uutils/coreutils/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22). +If you have any questions, feel free to ask them in the issues or on +[Discord](https://discord.gg/wQVJbvJ). ## Best practices @@ -40,13 +30,211 @@ search the issues to make sure no one else is working on it. ## Platforms -We take pride in supporting many operating systems and architectures. +We take pride in supporting many operating systems and architectures. Any code +you contribute must at least compile without warnings for all platforms in the +CI. However, you can use `#[cfg(...)]` attributes to create platform dependent features. -**Tip:** -For Windows, Microsoft provides some images (VMWare, Hyper-V, VirtualBox and Parallels) -for development: +**Tip:** For Windows, Microsoft provides some images (VMWare, Hyper-V, +VirtualBox and Parallels) for development: +## Tools + +We have an extensive CI that will check your code before it can be merged. This +section explains how to run those checks locally to avoid waiting for the CI. + +### pre-commit hooks + +A configuration for `pre-commit` is provided in the repository. It allows +automatically checking every git commit you make to ensure it compiles, and +passes `clippy` and `rustfmt` without warnings. + +To use the provided hook: + +1. [Install `pre-commit`](https://pre-commit.com/#install) +1. Run `pre-commit install` while in the repository directory + +Your git commits will then automatically be checked. If a check fails, an error +message will explain why, and your commit will be canceled. You can then make +the suggested changes, and run `git commit ...` again. + +### clippy + +```shell +cargo clippy --all-targets --all-features +``` + +The `msrv` key in the clippy configuration file `clippy.toml` is used to disable +lints pertaining to newer features by specifying the minimum supported Rust +version (MSRV). + +### rustfmt + +```shell +cargo fmt --all +``` + +### cargo-deny + +This project uses [cargo-deny](https://github.com/EmbarkStudios/cargo-deny/) to +detect duplicate dependencies, checks licenses, etc. To run it locally, first +install it and then run with: + +``` +cargo deny --all-features check all +``` + +### Markdown linter + +We use [markdownlint](https://github.com/DavidAnson/markdownlint) to lint the +Markdown files in the repository. + +### Spell checker + +We use `cspell` as spell checker for all files in the project. If you are using +VS Code, you can install the +[code spell checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker) +extension to enable spell checking within your editor. Otherwise, you can +install [cspell](https://cspell.org/) separately. + +If you want to make the spell checker ignore a word, you can add + +```rust +// spell-checker:ignore word_to_ignore +``` + +at the top of the file. + +## Testing + +Testing can be done using either Cargo or `make`. + +### Testing with Cargo + +Just like with building, we follow the standard procedure for testing using +Cargo: + +```shell +cargo test +``` + +By default, `cargo test` only runs the common programs. To run also platform +specific tests, run: + +```shell +cargo test --features unix +``` + +If you would prefer to test a select few utilities: + +```shell +cargo test --features "chmod mv tail" --no-default-features +``` + +If you also want to test the core utilities: + +```shell +cargo test -p uucore -p coreutils +``` + +To debug: + +```shell +gdb --args target/debug/coreutils ls +(gdb) b ls.rs:79 +(gdb) run +``` + +### Testing with GNU Make + +To simply test all available utilities: + +```shell +make test +``` + +To test all but a few of the available utilities: + +```shell +make SKIP_UTILS='UTILITY_1 UTILITY_2' test +``` + +To test only a few of the available utilities: + +```shell +make UTILS='UTILITY_1 UTILITY_2' test +``` + +To include tests for unimplemented behavior: + +```shell +make UTILS='UTILITY_1 UTILITY_2' SPEC=y test +``` + +### Run Busybox Tests + +This testing functionality is only available on *nix operating systems and +requires `make`. + +To run busybox tests for all utilities for which busybox has tests + +```shell +make busytest +``` + +To run busybox tests for a few of the available utilities + +```shell +make UTILS='UTILITY_1 UTILITY_2' busytest +``` + +To pass an argument like "-v" to the busybox test runtime + +```shell +make UTILS='UTILITY_1 UTILITY_2' RUNTEST_ARGS='-v' busytest +``` + +### Comparing with GNU + +To run uutils against the GNU test suite locally, run the following commands: + +```shell +bash util/build-gnu.sh +bash util/run-gnu-test.sh +# To run a single test: +bash util/run-gnu-test.sh tests/touch/not-owner.sh # for example +# To run several tests: +bash util/run-gnu-test.sh tests/touch/not-owner.sh tests/rm/no-give-up.sh # for example +# If this is a perl (.pl) test, to run in debug: +DEBUG=1 bash util/run-gnu-test.sh tests/misc/sm3sum.pl +``` + +Note that it relies on individual utilities (not the multicall binary). + +### Improving the GNU compatibility + +The Python script `./util/remaining-gnu-error.py` shows the list of failing +tests in the CI. + +To improve the GNU compatibility, the following process is recommended: + +1. Identify a test (the smaller, the better) on a program that you understand or + is easy to understand. You can use the `./util/remaining-gnu-error.py` script + to help with this decision. +1. Build both the GNU and Rust coreutils using: `bash util/build-gnu.sh` +1. Run the test with `bash util/run-gnu-test.sh ` +1. Start to modify `` to understand what is wrong. Examples: + 1. Add `set -v` to have the bash verbose mode + 1. Add `echo $?` where needed + 1. When the variable `fail` is used in the test, `echo $fail` to see when the + test started to fail + 1. Bump the content of the output (ex: `cat err`) + 1. ... +1. Or, if the test is simple, extract the relevant information to create a new + test case running both GNU & Rust implementation +1. Start to modify the Rust implementation to match the expected behavior +1. Add a test to make sure that we don't regress (our test suite is super quick) + ## Commit messages To help the project maintainers review pull requests from contributors across @@ -105,15 +293,33 @@ uutils: add new utility gitignore: add temporary files ``` -## cargo-deny +## Code coverage -This project uses [cargo-deny](https://github.com/EmbarkStudios/cargo-deny/) to -detect duplicate dependencies, checks licenses, etc. To run it locally, first -install it and then run with: + +Code coverage report can be generated using [grcov](https://github.com/mozilla/grcov). + +### Using Nightly Rust + +To generate [gcov-based](https://github.com/mozilla/grcov#example-how-to-generate-gcda-files-for-a-rust-project) coverage report + +```shell +export CARGO_INCREMENTAL=0 +export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort" +export RUSTDOCFLAGS="-Cpanic=abort" +cargo build # e.g., --features feat_os_unix +cargo test # e.g., --features feat_os_unix test_pathchk +grcov . -s . --binary-path ./target/debug/ -t html --branch --ignore-not-existing --ignore build.rs --excl-br-line "^\s*((debug_)?assert(_eq|_ne)?\#\[derive\()" -o ./target/debug/coverage/ +# open target/debug/coverage/index.html in browser ``` -cargo deny --all-features check all -``` + +if changes are not reflected in the report then run `cargo clean` and run the above commands. + +### Using Stable Rust + +If you are using stable version of Rust that doesn't enable code coverage instrumentation by default +then add `-Z-Zinstrument-coverage` flag to `RUSTFLAGS` env variable specified above. + ## Other implementations diff --git a/DEVELOPER_INSTRUCTIONS.md b/DEVELOPER_INSTRUCTIONS.md deleted file mode 100644 index 3641f57b9..000000000 --- a/DEVELOPER_INSTRUCTIONS.md +++ /dev/null @@ -1,68 +0,0 @@ -# Documentation - -The source of the documentation is available on: - - - -The documentation is updated everyday on this repository: - - - -## Running GNU tests - - - -- Check out next to your fork as gnu -- Check out next to your fork as gnulib -- Rename the checkout of your fork to uutils - -At the end you should have uutils, gnu and gnulib checked out next to each other. - -- Run `cd uutils && ./util/build-gnu.sh && cd ..` to get everything ready (this may take a while) -- Finally, you can run tests with `bash uutils/util/run-gnu-test.sh `. Instead of `` insert the tests you want to run, e.g. `tests/misc/wc-proc.sh`. - -## Code Coverage Report Generation - - - -Code coverage report can be generated using [grcov](https://github.com/mozilla/grcov). - -### Using Nightly Rust - -To generate [gcov-based](https://github.com/mozilla/grcov#example-how-to-generate-gcda-files-for-a-rust-project) coverage report - -```shell -export CARGO_INCREMENTAL=0 -export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort" -export RUSTDOCFLAGS="-Cpanic=abort" -cargo build # e.g., --features feat_os_unix -cargo test # e.g., --features feat_os_unix test_pathchk -grcov . -s . --binary-path ./target/debug/ -t html --branch --ignore-not-existing --ignore build.rs --excl-br-line "^\s*((debug_)?assert(_eq|_ne)?\#\[derive\()" -o ./target/debug/coverage/ -# open target/debug/coverage/index.html in browser -``` - -if changes are not reflected in the report then run `cargo clean` and run the above commands. - -### Using Stable Rust - -If you are using stable version of Rust that doesn't enable code coverage instrumentation by default -then add `-Z-Zinstrument-coverage` flag to `RUSTFLAGS` env variable specified above. - -## pre-commit hooks - -A configuration for `pre-commit` is provided in the repository. It allows automatically checking every git commit you make to ensure it compiles, and passes `clippy` and `rustfmt` without warnings. - -To use the provided hook: - -1. [Install `pre-commit`](https://pre-commit.com/#install) -1. Run `pre-commit install` while in the repository directory - -Your git commits will then automatically be checked. If a check fails, an error message will explain why, and your commit will be canceled. You can then make the suggested changes, and run `git commit ...` again. - -## Using Clippy - -The `msrv` key in the clippy configuration file `clippy.toml` is used to disable lints pertaining to newer features by specifying the minimum supported Rust version (MSRV). However, this key is only supported on `nightly`. To invoke clippy without errors, use `cargo +nightly clippy`. In order to also check tests and non-default crate features, use `cargo +nightly clippy --all-targets --all-features`. - -## Markdown linter - -We use to lint the Markdown files. diff --git a/README.md b/README.md index 97e115a6a..8b57855b5 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,10 @@ + + + +
+ +![uutils logo](docs/src/logo.svg) + # uutils coreutils [![Crates.io](https://img.shields.io/crates/v/coreutils.svg)](https://crates.io/crates/coreutils) @@ -9,15 +16,14 @@ [![CodeCov](https://codecov.io/gh/uutils/coreutils/branch/master/graph/badge.svg)](https://codecov.io/gh/uutils/coreutils) ![MSRV](https://img.shields.io/badge/MSRV-1.64.0-brightgreen) ------------------------------------------------ +
- - +--- -uutils is an attempt at writing universal (as in cross-platform) CLI -utilities in [Rust](http://www.rust-lang.org). -While all programs have been implemented, some options might be missing -or different behavior might be experienced. + +uutils coreutils is a cross-platform reimplementation of the GNU coreutils in +[Rust](http://www.rust-lang.org). While all programs have been implemented, some +options might be missing or different behavior might be experienced. To install it: @@ -27,13 +33,15 @@ cargo install coreutils ``` -## Why? -uutils aims to work on as many platforms as possible, to be able to use the -same utils on Linux, Mac, Windows and other platforms. This ensures, for -example, that scripts can be easily transferred between platforms. Rust was -chosen not only because it is fast and safe, but is also excellent for -writing cross-platform code. +## Goals + +uutils aims to be a drop-in replacement for the GNU utils. Differences with GNU +are treated as bugs. + +uutils aims to work on as many platforms as possible, to be able to use the same +utils on Linux, Mac, Windows and other platforms. This ensures, for example, +that scripts can be easily transferred between platforms. ## Documentation @@ -42,10 +50,11 @@ uutils has both user and developer documentation available: - [User Manual](https://uutils.github.io/user/) - [Developer Documentation](https://uutils.github.io/dev/coreutils/) -Both can also be generated locally, the instructions for that can be found in the -[coreutils docs](https://github.com/uutils/uutils.github.io) repository. +Both can also be generated locally, the instructions for that can be found in +the [coreutils docs](https://github.com/uutils/uutils.github.io) repository. + ## Requirements - Rust (`cargo`, `rustc`) @@ -53,13 +62,13 @@ Both can also be generated locally, the instructions for that can be found in th ### Rust Version -uutils follows Rust's release channels and is tested against stable, beta and nightly. -The current Minimum Supported Rust Version (MSRV) is `1.64.0`. +uutils follows Rust's release channels and is tested against stable, beta and +nightly. The current Minimum Supported Rust Version (MSRV) is `1.64.0`. ## Building -There are currently two methods to build the uutils binaries: either Cargo -or GNU Make. +There are currently two methods to build the uutils binaries: either Cargo or +GNU Make. > Building the full package, including all documentation, requires both Cargo > and Gnu Make on a Unix platform. @@ -73,8 +82,8 @@ cd coreutils ### Cargo -Building uutils using Cargo is easy because the process is the same as for -every other Rust program: +Building uutils using Cargo is easy because the process is the same as for every +other Rust program: ```shell cargo build --release @@ -83,9 +92,9 @@ cargo build --release This command builds the most portable common core set of uutils into a multicall (BusyBox-type) binary, named 'coreutils', on most Rust-supported platforms. -Additional platform-specific uutils are often available. Building these -expanded sets of uutils for a platform (on that platform) is as simple as -specifying it as a feature: +Additional platform-specific uutils are often available. Building these expanded +sets of uutils for a platform (on that platform) is as simple as specifying it +as a feature: ```shell cargo build --release --features macos @@ -96,18 +105,18 @@ cargo build --release --features unix ``` If you don't want to build every utility available on your platform into the -final binary, you can also specify which ones you want to build manually. -For example: +final binary, you can also specify which ones you want to build manually. For +example: ```shell cargo build --features "base32 cat echo rm" --no-default-features ``` -If you don't want to build the multicall binary and would prefer to build -the utilities as individual binaries, that is also possible. Each utility -is contained in its own package within the main repository, named -"uu_UTILNAME". To build individual utilities, use cargo to build just the -specific packages (using the `--package` [aka `-p`] option). For example: +If you don't want to build the multicall binary and would prefer to build the +utilities as individual binaries, that is also possible. Each utility is +contained in its own package within the main repository, named "uu_UTILNAME". To +build individual utilities, use cargo to build just the specific packages (using +the `--package` [aka `-p`] option). For example: ```shell cargo build -p uu_base32 -p uu_cat -p uu_echo -p uu_rm @@ -151,10 +160,12 @@ Likewise, installing can simply be done using: cargo install --path . --locked ``` -This command will install uutils into Cargo's *bin* folder (*e.g.* `$HOME/.cargo/bin`). +This command will install uutils into Cargo's _bin_ folder (_e.g._ +`$HOME/.cargo/bin`). -This does not install files necessary for shell completion or manpages. -For manpages or shell completion to work, use `GNU Make` or see `Manually install shell completions`/`Manually install manpages`. +This does not install files necessary for shell completion or manpages. For +manpages or shell completion to work, use `GNU Make` or see +`Manually install shell completions`/`Manually install manpages`. ### Install with GNU Make @@ -207,8 +218,8 @@ be generated; See `Manually install shell completions`. ### Manually install shell completions -The `coreutils` binary can generate completions for the `bash`, `elvish`, `fish`, `powershell` -and `zsh` shells. It prints the result to stdout. +The `coreutils` binary can generate completions for the `bash`, `elvish`, +`fish`, `powershell` and `zsh` shells. It prints the result to stdout. The syntax is: @@ -216,8 +227,8 @@ The syntax is: cargo run completion ``` -So, to install completions for `ls` on `bash` to `/usr/local/share/bash-completion/completions/ls`, -run: +So, to install completions for `ls` on `bash` to +`/usr/local/share/bash-completion/completions/ls`, run: ```shell cargo run completion ls bash > /usr/local/share/bash-completion/completions/ls @@ -226,12 +237,12 @@ cargo run completion ls bash > /usr/local/share/bash-completion/completions/ls ### Manually install manpages To generate manpages, the syntax is: + ```bash cargo run manpage ``` -So, to install the manpage for `ls` to `/usr/local/share/man/man1/ls.1` -run: +So, to install the manpage for `ls` to `/usr/local/share/man/man1/ls.1` run: ```bash cargo run manpage ls > /usr/local/share/man/man1/ls.1 @@ -239,8 +250,8 @@ cargo run manpage ls > /usr/local/share/man/man1/ls.1 ## Un-installation -Un-installation differs depending on how you have installed uutils. If you used -Cargo to install, use Cargo to uninstall. If you used GNU Make to install, use +Un-installation differs depending on how you have installed uutils. If you used +Cargo to install, use Cargo to uninstall. If you used GNU Make to install, use Make to uninstall. ### Uninstall with Cargo @@ -280,245 +291,21 @@ make PREFIX=/my/path uninstall -## Testing - -Testing can be done using either Cargo or `make`. - -### Testing with Cargo - -Just like with building, we follow the standard procedure for testing using -Cargo: - -```shell -cargo test -``` - -By default, `cargo test` only runs the common programs. To run also platform -specific tests, run: - -```shell -cargo test --features unix -``` - -If you would prefer to test a select few utilities: - -```shell -cargo test --features "chmod mv tail" --no-default-features -``` - -If you also want to test the core utilities: - -```shell -cargo test -p uucore -p coreutils -``` - -To debug: - -```shell -gdb --args target/debug/coreutils ls -(gdb) b ls.rs:79 -(gdb) run -``` - -### Testing with GNU Make - -To simply test all available utilities: - -```shell -make test -``` - -To test all but a few of the available utilities: - -```shell -make SKIP_UTILS='UTILITY_1 UTILITY_2' test -``` - -To test only a few of the available utilities: - -```shell -make UTILS='UTILITY_1 UTILITY_2' test -``` - -To include tests for unimplemented behavior: - -```shell -make UTILS='UTILITY_1 UTILITY_2' SPEC=y test -``` - -### Run Busybox Tests - -This testing functionality is only available on *nix operating systems and -requires `make`. - -To run busybox tests for all utilities for which busybox has tests - -```shell -make busytest -``` - -To run busybox tests for a few of the available utilities - -```shell -make UTILS='UTILITY_1 UTILITY_2' busytest -``` - -To pass an argument like "-v" to the busybox test runtime - -```shell -make UTILS='UTILITY_1 UTILITY_2' RUNTEST_ARGS='-v' busytest -``` - -### Comparing with GNU +## GNU test suite compatibility Below is the evolution of how many GNU tests uutils passes. A more detailed breakdown of the GNU test results of the main branch can be found [in the user manual](https://uutils.github.io/user/test_coverage.html). +See for the main meta bugs +(many are missing). + ![Evolution over time](https://github.com/uutils/coreutils-tracking/blob/main/gnu-results.png?raw=true) -To run locally: - -```shell -bash util/build-gnu.sh -bash util/run-gnu-test.sh -# To run a single test: -bash util/run-gnu-test.sh tests/touch/not-owner.sh # for example -# To run several tests: -bash util/run-gnu-test.sh tests/touch/not-owner.sh tests/rm/no-give-up.sh # for example -# If this is a perl (.pl) test, to run in debug: -DEBUG=1 bash util/run-gnu-test.sh tests/misc/sm3sum.pl -``` - -Note that it relies on individual utilities (not the multicall binary). - -### Improving the GNU compatibility - -The Python script `./util/remaining-gnu-error.py` shows the list of failing tests in the CI. - -To improve the GNU compatibility, the following process is recommended: - -1. Identify a test (the smaller, the better) on a program that you understand or is easy to understand. You can use the `./util/remaining-gnu-error.py` script to help with this decision. -1. Build both the GNU and Rust coreutils using: `bash util/build-gnu.sh` -1. Run the test with `bash util/run-gnu-test.sh ` -1. Start to modify `` to understand what is wrong. Examples: - 1. Add `set -v` to have the bash verbose mode - 1. Add `echo $?` where needed - 1. When the variable `fail` is used in the test, `echo $fail` to see when the test started to fail - 1. Bump the content of the output (ex: `cat err`) - 1. ... -1. Or, if the test is simple, extract the relevant information to create a new test case running both GNU & Rust implementation -1. Start to modify the Rust implementation to match the expected behavior -1. Add a test to make sure that we don't regress (our test suite is super quick) - ## Contributing To contribute to uutils, please see [CONTRIBUTING](CONTRIBUTING.md). -## Utilities - -Please note that this is not fully accurate: - -- Some new options can be added / removed in the GNU implementation; -- Some error management might be missing; -- Some behaviors might be different. - -See for the main meta bugs -(many are missing). - -| Done | WIP | -|-----------|-----------| -| arch | cp | -| base32 | date | -| base64 | dd | -| basename | df | -| basenc | expr | -| cat | install | -| chcon | ls | -| chgrp | more | -| chmod | numfmt | -| chown | od (`--strings` and 128-bit data types missing) | -| chroot | pr | -| cksum | printf | -| comm | sort | -| csplit | split | -| cut | tac | -| dircolors | test | -| dirname | dir | -| du | vdir | -| echo | stty | -| env | | -| expand | | -| factor | | -| false | | -| fmt | | -| fold | | -| groups | | -| hashsum | | -| head | | -| hostid | | -| hostname | | -| id | | -| join | | -| kill | | -| link | | -| ln | | -| logname | | -| ~~md5sum~~ (replaced by [hashsum](https://github.com/uutils/coreutils/blob/main/src/uu/hashsum/src/hashsum.rs)) | | -| ~~sha1sum~~ (replaced by [hashsum](https://github.com/uutils/coreutils/blob/main/src/uu/hashsum/src/hashsum.rs)) | | -| ~~sha224sum~~ (replaced by [hashsum](https://github.com/uutils/coreutils/blob/main/src/uu/hashsum/src/hashsum.rs)) | | -| ~~sha256sum~~ (replaced by [hashsum](https://github.com/uutils/coreutils/blob/main/src/uu/hashsum/src/hashsum.rs)) | | -| ~~sha384sum~~ (replaced by [hashsum](https://github.com/uutils/coreutils/blob/main/src/uu/hashsum/src/hashsum.rs)) | | -| ~~sha512sum~~ (replaced by [hashsum](https://github.com/uutils/coreutils/blob/main/src/uu/hashsum/src/hashsum.rs)) | | -| mkdir | | -| mkfifo | | -| mknod | | -| mktemp | | -| mv | | -| nice | | -| nl | | -| nohup | | -| nproc | | -| paste | | -| pathchk | | -| pinky | | -| printenv | | -| ptx | | -| pwd | | -| readlink | | -| realpath | | -| relpath | | -| rm | | -| rmdir | | -| runcon | | -| seq | | -| shred | | -| shuf | | -| sleep | | -| stat | | -| stdbuf | | -| sum | | -| sync | | -| tail | | -| tee | | -| timeout | | -| touch | | -| tr | | -| true | | -| truncate | | -| tsort | | -| tty | | -| uname | | -| unexpand | | -| uniq | | -| unlink | | -| uptime | | -| users | | -| wc | | -| who | | -| whoami | | -| yes | | - ## License uutils is licensed under the MIT License - see the `LICENSE` file for details