ops leaking secrets if `subprocess.CalledProcessError` happens with a `secret-*` CLI command

### Summary The issue here is that we pass the secret content as one of the args via CLI. This issue may affect any of our charms that are using: Juju (>=3.0), Juju secrets and not correctly capturing and processing `subprocess.CalledProcessError`. There are two points that may log this command, in different files: First, if there is an error during a secret handling, there will be a `subprocess.CalledProcessError`, which will contain the CLI comand + all its args. This is going to be logged in any logging level. This exception, if not caught by the charm, will bubble up to the `/var/log/juju/` logs and syslog journal. Now, on Ubuntu 22.04, these logs are protected with: ``` $ juju ssh -m controller 0 -- ls -la /var/log/juju/ total 224 drwxr-xr-x 2 syslog adm 4096 Jul 14 10:59 . drwxrwxr-x 9 root syslog 4096 Jul 14 10:58 .. -rw-r----- 1 syslog adm 20124 Jul 14 11:10 audit.log -rw-r----- 1 syslog adm 110432 Jul 14 11:10 logsink.log -rw-r----- 1 syslog adm 80783 Jul 14 11:06 machine-0.log -rw-r----- 1 syslog adm 766 Jul 14 11:10 machine-lock.log -rw-r--r-- 1 root root 0 Jul 14 10:59 slow-query.log -rw-r----- 1 syslog adm 3732 Jul 14 11:10 unit-controller-0.log ``` Second, certain audit setups may log terminal commands, which would result in this command being logged with its secrets. It is unknown if this is done on ubuntu security benchmarks, such as CIS hardening. Keep in mind these logs may be copied or even backed up. Which exposes it to more services in the user's environment (e.g. CI runs in GH - although these are dummy password generated per test only). Passing secrets straight via CLI is not advised. Here are some ways out: 1) Redacting: which commands and which args represent secrets are known, so they can be redacted. It would also mean capturing a `subprocess.CalledProcessError`, redacting its content and reissuing the same type of exception; this will not cover the case `auditd` is set to log CLI commands, if that is a risk 2) Temp files: secret-add, for example, can use a secret file instead, [as can be seen here](https://github.com/juju/juju/blob/20b7d944acc985ce91419728529d5fb2abd14d68/cmd/juju/secrets/add.go#L62). However, if ops uses a file, ops will need to be sure to correctly remove it later 3) stdin: not sure it is accepted by secret-* commands, but generally, secrets are not shown on CLI whilst typing them; auditd may not capture that stdin ### Severity Rationale This is a [CWE-532](https://cwe.mitre.org/data/definitions/532.html). Potentially, these secrets can lead to privilege escalation but Ubuntu default is to have logs only accessible to `adm` group users. Marking this issue as "Moderate", as this report is not presenting a clear way on how to get access to the logs themselves: either getting local access to an `adm` group user (e.g. ubuntu) or recovering logs stored on a 3rd party service. ### Details ``` 2024-07-12T14:27:58.0175389Z unit-opensearch-3: 14:27:53 ERROR unit.opensearch/3.juju-log certificates:11: Uncaught exception while in charm code: 2024-07-12T14:27:58.0175524Z Traceback (most recent call last): 2024-07-12T14:27:58.0175957Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/model.py", line 3180, in _run 2024-07-12T14:27:58.0176165Z result = subprocess.run(args, **kwargs) # type: ignore 2024-07-12T14:27:58.0176381Z File "/usr/lib/python3.10/subprocess.py", line 526, in run 2024-07-12T14:27:58.0176561Z raise CalledProcessError(retcode, process.args, 2024-07-12T14:27:58.0177749Z subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-opensearch-3/secret-add', '--label', 'opensearch:app:app-admin', '--owner', 'application', 'keystore-password-ca=aUE...')' returned non-zero exit status 1. 2024-07-12T14:27:58.0177765Z 2024-07-12T14:27:58.0178023Z The above exception was the direct cause of the following exception: 2024-07-12T14:27:58.0178038Z 2024-07-12T14:27:58.0178168Z Traceback (most recent call last): 2024-07-12T14:27:58.0178588Z File "/var/lib/juju/agents/unit-opensearch-3/charm/./src/charm.py", line 213, in <module> 2024-07-12T14:27:58.0178724Z main(OpenSearchOperatorCharm) 2024-07-12T14:27:58.0179129Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/main.py", line 548, in main 2024-07-12T14:27:58.0179237Z manager.run() 2024-07-12T14:27:58.0179640Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/main.py", line 527, in run 2024-07-12T14:27:58.0179745Z self._emit() 2024-07-12T14:27:58.0180150Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/main.py", line 516, in _emit 2024-07-12T14:27:58.0180359Z _emit_charm_event(self.charm, self.dispatcher.event_name) 2024-07-12T14:27:58.0180840Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/main.py", line 147, in _emit_charm_event 2024-07-12T14:27:58.0180974Z event_to_emit.emit(*args, **kwargs) 2024-07-12T14:27:58.0181537Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 348, in emit 2024-07-12T14:27:58.0181656Z framework._emit(event) 2024-07-12T14:27:58.0182091Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 860, in _emit 2024-07-12T14:27:58.0182211Z self._reemit(event_path) 2024-07-12T14:27:58.0182659Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 950, in _reemit 2024-07-12T14:27:58.0182770Z custom_handler(event) 2024-07-12T14:27:58.0183492Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/tls_certificates_interface/v3/tls_certificates.py", line 1811, in _on_relation_changed 2024-07-12T14:27:58.0183743Z self.on.certificate_available.emit( 2024-07-12T14:27:58.0184173Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 348, in emit 2024-07-12T14:27:58.0184291Z framework._emit(event) 2024-07-12T14:27:58.0184717Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 860, in _emit 2024-07-12T14:27:58.0184887Z self._reemit(event_path) 2024-07-12T14:27:58.0186171Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 950, in _reemit 2024-07-12T14:27:58.0186306Z custom_handler(event) 2024-07-12T14:27:58.0187043Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_tls.py", line 209, in _on_certificate_available 2024-07-12T14:27:58.0187320Z self.store_new_ca(self.charm.secrets.get_object(scope, cert_type.val)) 2024-07-12T14:27:58.0187942Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_tls.py", line 444, in store_new_ca 2024-07-12T14:27:58.0188242Z self._create_keystore_pwd_if_not_exists(Scope.APP, CertType.APP_ADMIN, "ca") 2024-07-12T14:27:58.0188981Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_tls.py", line 432, in _create_keystore_pwd_if_not_exists 2024-07-12T14:27:58.0189119Z self.charm.secrets.put_object( 2024-07-12T14:27:58.0189738Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_secrets.py", line 359, in put_object 2024-07-12T14:27:58.0189944Z self._add_or_update_juju_secret(scope, key, value, merge) 2024-07-12T14:27:58.0190652Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_secrets.py", line 272, in _add_or_update_juju_secret 2024-07-12T14:27:58.0190815Z return self._add_juju_secret(scope, key, value) 2024-07-12T14:27:58.0191462Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_secrets.py", line 228, in _add_juju_secret 2024-07-12T14:27:58.0191660Z secret = scope_obj.add_secret(safe_value, label=label) 2024-07-12T14:27:58.0192110Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/model.py", line 477, in add_secret 2024-07-12T14:27:58.0192241Z id = self._backend.secret_add( 2024-07-12T14:27:58.0192686Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/model.py", line 3624, in secret_add 2024-07-12T14:27:58.0192948Z result = self._run('secret-add', *args, return_output=True) 2024-07-12T14:27:58.0193356Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/model.py", line 3182, in _run 2024-07-12T14:27:58.0193489Z raise ModelError(e.stderr) from e 2024-07-12T14:27:58.0193685Z ops.model.ModelError: ERROR this unit is not the leader 2024-07-12T14:27:58.0193692Z ``` From CI: https://github.com/canonical/opensearch-operator/actions/runs/9908987369/job/27376377521?pr=364 ### PoC 1) Deploy anything with juju 2) Run a dummy secret-add call that will fail 3) See the uncaught subprocess error ### Impact Juju secrets are generally composed of private keys, passwords, etc; generally valuable credentials that, if leaked, will likely allow an attacker to get privileged access to its target or other targets in the environment.

Quoted source text, attributed separately from HOL analysis.