Add cli option to publish to core db. #744

anders-ahlberg · 2025-06-24T04:25:10Z

This depends on changes in pending pull request #743 (signing and verifying).

…lacks provider section.

anlu85 · 2025-09-12T08:08:13Z

fusesoc/main.py

+            print("Aborted.")
+            return False
+
+    target = uri + "/v1/publish/"


The APIs entry point is at /api, so the full path /api/v1/publish/ (see Publish a core file).
Seems like the /api part is missing in the target generated URL.

I choose to put the api part in the publish-uri setting (e.g. publish-uri = https://fscpd.qamcom.se/api). I thought it made more sense to let that that point to the root of the api, and then let the code build in that.

anlu85 · 2025-09-12T08:15:37Z

I think it would be great to have an additional CLI interface to validate a core using the server’s API endpoint (see Validate a core file).
This would make it easy to check whether a core satisfies the server’s requirements without actually publishing it.

olofk · 2025-09-19T06:26:39Z

fusesoc/main.py

+        sf_data = None
+    if pem:
+        print("with certificate from: " + pem)
+    print("to api at: " + uri)


This fails if uri is not defined

Do we want a hardcoded default uri? If so, what should it point to?

If you like to add a default, pointing to the "official" FOSSI Foundation server instance might be a good option:
https://fusesoc.fossi-foundation.net/

olofk · 2025-09-19T06:27:31Z

We need some documentation as well.

olofk · 2025-09-19T06:30:47Z

fusesoc/main.py

+def core_publish(fs, args):
+    core = _get_core(fs, args.core)
+    uri = fs.config.publish_uri
+    pem = fs.config.publish_uri_pem


Why do we need this?

Do you mean specifically the pem option? It was needed during development since that server was self signed, and why not keep the option?

Allowing HTTPS connections with self-signed certificates - or even HTTP connections - could be a useful feature for local deployments or testing of the FuseSoC Package Directory Server.

olofk · 2025-09-19T06:34:46Z

fusesoc/main.py

+    res = requests.post(target, files=body, allow_redirects=True, verify=pem)
+    if not res.ok:
+        print("Request returned http result", res.status_code, res.reason)
+        err = json.loads(res.content)


Need to catch more errors here. If the server responds 404 it looks like there isn't a valid json reply at all

Request returned http result 404 Not Found Traceback (most recent call last): File "/home/olof/projects/serv/bin/fusesoc", line 8, in <module> sys.exit(main()) ^^^^^^ File "/home/olof/projects/fusesoc/fusesoc/fusesoc/main.py", line 950, in main fusesoc(args) File "/home/olof/projects/fusesoc/fusesoc/fusesoc/main.py", line 940, in fusesoc args.func(fs, args) File "/home/olof/projects/fusesoc/fusesoc/fusesoc/main.py", line 417, in core_publish err = json.loads(res.content) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/json/__init__.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

What is a good server to test this against today without causing havoc? I have not been keeping up with recent deployments and deprecations.

Our internal test server is still up an running.
Alternative you can spin up your own test server following FuseSoC Package Directory Quick Start or Development Guide.

olofk · 2025-09-19T06:58:45Z

fusesoc/main.py

+    if not res.ok:
+        print("Request returned http result", res.status_code, res.reason)
+        err = json.loads(res.content)
+        print(json.dumps(err, indent=4))


Let's try to handle known errors, e.g. validation errors, in a more user-friendly way. Instead of

Request returned http result 400 Bad Request { "non_field_errors": [ "Validation error in core::: 'description' is a required property" ] }

we could look for non_field_errors and write something like Validation error: 'description' is a required property

This seems to be just the forwarded error messages from the FuseSoC-PD Server. So I guess it is more an server side issue.
On server side, there are already some mechanism in place to replace the default error messages from core parsing/validation with custom messages (but there aren't much implemented right now, see end of core_directory/serializers.py)

olofk · 2025-09-19T07:10:32Z

fusesoc/main.py

    print(f"{sigfile} created")


+def guess_provider():


Since there is quite a lot of new code, I would prefer to have this in a separate publish.py. main.py is already way too large for its own good.

olofk · 2025-09-19T07:13:09Z

fusesoc/main.py

+            )
+            print(provider_info["yaml"])
+            return False
+        print("Adding the following provider section to the core file.")


We want to avoid adding the provider section to the .core file in the repo. Let's copy the core file to a temporary file instead where we can add the section and publish

olofk · 2025-09-19T07:46:35Z

fusesoc/main.py

+def guess_provider():
+    guess = {"found": False}
+    cmd = ["git", "remote", "-v"]
+    res = subprocess.run(cmd, capture_output=True).stdout.decode("utf-8").strip()


Let's run this from the root directory of the core instead. We can pick up the directory from core.core_root in core_publish and send it to guess_provider as an argument. The reason is that most FuseSoC users have a workspace which is separate from the directories where the cores are.

olofk · 2025-09-19T08:16:13Z

fusesoc/main.py

+    if len(fetchlines) < 1:
+        return guess
+    comps = fetchlines[0].split("/")
+    if comps[2] == "github.com":


This fails when there are more than one remote, e g.

$ git remote -v fork [email protected]:olofk/common_cells.git (fetch) fork [email protected]:olofk/common_cells.git (push) origin https://github.com/pulp-platform/common_cells (fetch) origin https://github.com/pulp-platform/common_cells (push)

In this particular case we might detect this situation and query the user, but my gut feeling tells me that there might be several more cases like this, so I think it would be good to have CLI options to manually specify at least the upstream URI

anders-ahlberg added 3 commits September 4, 2025 07:16

Add command to publish core (and optionally signature) files.

32c3409

Publishing can now talk to server, but server does not like our names.

155f164

Show publish error json prettier.

c460517

anders-ahlberg force-pushed the publish branch from 4a229af to c460517 Compare September 4, 2025 05:24

anders-ahlberg added 3 commits September 4, 2025 07:25

Added command parser that was accidentally removed in rebase.

655b56e

Added missing dependency on requests.

0327190

Only allow publishing of github repos, and offer to guess it if core …

a0c6922

…lacks provider section.

anlu85 reviewed Sep 12, 2025

View reviewed changes

olofk reviewed Sep 19, 2025

View reviewed changes

anlu85 mentioned this pull request Sep 23, 2025

Publishing core from GitHub through web interface fusesoc/fusesoc-webserver#6

Closed

5 tasks

Add cli option to publish to core db. #744

Are you sure you want to change the base?

Add cli option to publish to core db. #744

Uh oh!

Conversation

anders-ahlberg commented Jun 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anlu85 commented Sep 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olofk commented Sep 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!