`handle-generic-xml` cant handle nested `record` tags of different namespaces in one xml

[TobiasNx]

The problem was first spottet when harvesting oai pmh marcxml:

[blackwinter]

Of course it does, that's what pretty-printing is supposed to do. But it appears that the documents are nested incorrectly. Please post your workflow so that we can analyze at which step the error is introduced.

[Phu2]

Please post your workflow so that we can analyze at which step the error is introduced.

See my comment here metafacture/metafacture-playground#209 (comment)

[blackwinter]

Thanks for the workflow. This confirms my suspicion that the underlying issue isn't caused by the pretty-printing, or even the JSON encoder for that matter.

The problem is that the OAI-PMH opener yields XML documents with nested <record> elements (the first as child of <ListRecords> with namespace http://www.openarchives.org/OAI/2.0/, the second as child of <metadata> with namespace http://www.loc.gov/MARC21/slim) which the generic XML handler doesn't differentiate as it only looks at the local name, not at the namespace. So it starts a new record on the first element's opening tag, but closes the record on the second element's ending tag.

As a consequence, the records are nested inside each other. Which then also leads to excessive indentation when pretty-printing.

[Phu2]

Thanks for your analysis! 💡

the second as child of with namespace http://www.loc.gov/MARC21/slim)

this is specific to MARC21 of course.

Unfortunately i tested with oai_dc, that's why i came to conclusion that

processing records in 2 subsets also works. First subset using dateUntil="2020-01-01T00:00:00Z" and dateFrom="2020-01-01T00:00:00Z"for the second subset

which is misleading!

which the generic XML handler doesn't differentiate as it only looks at the local name, not at the namespace.

I don't think this is a good idea at all!

[blackwinter]

this is specific to MARC21 of course.

Exactly. Which is what your workflow specified.

I don't think this is a good idea at all!

Agreed :)

[TobiasNx]

so the additional none pretty printing typical line breaks are multiple indentations?

[blackwinter]

There are no additional line breaks. Are you talking about the line-wrapping of your file viewer (console) application?

[TobiasNx]

yes i was referring to the visual line-break which seems to be the result of line-wrapping

[Phu2]

I wonder why namespaces are omitted by default, see https://github.com/metafacture/metafacture-core/blob/master/metafacture-xml/src/main/java/org/metafacture/xml/GenericXmlHandler.java#L49 and https://github.com/metafacture/metafacture-core/blob/master/metafacture-xml/src/main/java/org/metafacture/xml/GenericXmlHandler.java#L135C66-L135C73 Any ideas?

[TobiasNx]

Historically namespaces were not included at all. It was added in 2021 #377

Also the record tag name configuration is currently not looking at namespaces even if the namespaces are kept: #544

[TobiasNx]

Coming from @blackwinter explanation, I try to show it with an example: The problem seems to result from a combination of requesting marc data from oai-pmh and metafactures handle-generic-xml:

"https://library.oapen.org/oai/request"
| open-oaipmh(dateFrom="2025-05-15",metadataPrefix="marc")
| decode-xml
| handle-generic-xml // does not matter to add(emitnamespace="true")
| encode-json(prettyPrinting="false")
| print;

The harvested oai/marc data looks like this:

<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="static/style.xsl"?>
<OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd">
	<responseDate>2025-05-16T07:56:16Z</responseDate>
	<request verb="ListRecords" metadataPrefix="marc">http://library.oapen.org/oai/request</request>
	<ListRecords>
		<record>
			<header>
...
			</header>
			<metadata>
				<record xmlns="http://www.loc.gov/MARC21/slim"
					xmlns:doc="http://www.lyncode.com/xoai"
					xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
					xmlns:dcterms="http://purl.org/dc/terms/" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd">
...
				</record>
			</metadata>
		</record>
		<record>
			<header status="deleted">
...
			</header>
		</record>
		<record>
			<header>
...
			</header>
			<metadata>
				<record xmlns="http://www.loc.gov/MARC21/slim"
					xmlns:doc="http://www.lyncode.com/xoai"
					xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
					xmlns:dcterms="http://purl.org/dc/terms/" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd">
...
				</record>
			</metadata>
		</record>
		<record>
			<header>
...
			</header>
			<metadata>
				<record xmlns="http://www.loc.gov/MARC21/slim"
					xmlns:doc="http://www.lyncode.com/xoai"
					xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
					xmlns:dcterms="http://purl.org/dc/terms/" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd">
...
				</record>
			</metadata>
		</record>
	</ListRecords>
</OAI-PMH>

The record tags of marc within the record tag of oai results in data garbage coming out of handle-generic-xml.

<record> (open oai record1)
    <record> (open marc record1)
    <record> (close marc record1)
<record> (close oai record1)
<record> (open oai record2)
    <record> (open marc record2)
    <record> (close marc record2)
<record> (close oai record2)
<record> (open oai record3)
    <record> (open marc record3)
    <record> (close marc record3)
<record> (close oai record3)

The problem results from: oai record1 is closed by marc record1, marc record1 is closed by oai record1 etc. this creates confusion for metafacture.

not pretty printed json:

{"header":{"status":"deleted","identifier":{"value":"oai:library.oapen.org:20.500.12657/24255"},"datestamp":{"value":"2025-05-15T11:23:58Z"},"setSpec":{"value":"com_20.500.12657_5"},"setSpec":{"value":"col_20.500.12657_6"}}}
{"header":{"identifier":{"value":"oai:library.oapen.org:20.500.12657/101622"},"datestamp":{"value":"2025-05-15T07:12:23Z"},"setSpec":{"value":"com_20.500.12657_5"},"setSpec":{"value":"col_20.500.12657_6"}},"metadata":{"record":{"xsi:schemaLocation":"http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd","leader":{"value":"00925njm 22002777a 4500"},"datafield":{"ind1":" ","ind2":" ","tag":"042","subfield":{"code":"a","value":"dc"}},"datafield":{"ind1":" ","ind2":" ","tag":"720","subfield":{"code":"a","value":"Bladini, Moa"},"subfield":{"code":"e","value":"author"}},"datafield":{"ind1":" ","ind2":" ","tag":"720","subfield":{"code":"a","value":"Svensson, Eva-Maria"},"subfield":{"code":"e","value":"author"}},"datafield":{"ind1":" ","ind2":" ","tag":"260","subfield":{"code":"c","value":"2025"}},"datafield":{"ind1":" ","ind2":" ","tag":"520","subfield":{"code":"a","value":"Providing an innovative approach to understanding and addressing anti-gender hate speech, this book focuses on its effects on everyday life, how it is handled within legal frameworks, and how it challenges democratic principles. Unlike previous research, which is often limited to currently regulated forms of hate speech, this book is focused on gender-based hate speech, demonstrating its broader social and ideological impacts. By contextualising freedom of expression within democratic values, it offers a new framework for addressing anti-gender hate speech as a threat to both individual dignity and societal diversity. Through this lens, the authors highlight how anti-gender hate speech represents not only a breach of individual rights but also a structural challenge to democracy, requiring responses that reflect these dual dimensions. The book provides a unique contribution to both legal scholarship and democratic theory by rethinking the balance between gender equality and freedom of expression as cooperative as well as inherently contextual and inclusive. Rethinking Online Anti-Gender Hate Speech will be of great interest to students and scholars of Criminology, Law, Media and Communication, Sociology, and Women’s and Gender Studies."}},"datafield":{"ind1":"8","ind2":" ","tag":"024","subfield":{"code":"a","value":"https://library.oapen.org/handle/20.500.12657/101622"}},"datafield":{"ind1":"0","ind2":"0","tag":"245","subfield":{"code":"a","value":"Rethinking Online Anti-Gender Hate Speech"}}}
,"":{"header":{"identifier":{"value":"oai:library.oapen.org:20.500.12657/101821"},"datestamp":{"value":"2025-05-16T01:26:09Z"},"setSpec":{"value":"com_20.500.12657_5"},"setSpec":{"value":"col_20.500.12657_6"}},"metadata":{"record":{"xsi:schemaLocation":"http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd","leader":{"value":"00925njm 22002777a 4500"},"datafield":{"ind1":" ","ind2":" ","tag":"042","subfield":{"code":"a","value":"dc"}},"datafield":{"ind1":" ","ind2":" ","tag":"260","subfield":{"code":"c","value":"2018"}},"datafield":{"ind1":" ","ind2":" ","tag":"520","subfield":{"code":"a","value":"The Open Access version of this book, available at http://www.tandfebooks.com/doi/view/10.4324/9781315180250, has been made available under a Creative Commons Attribution-Non Commercial-No Derivatives 4.0 license Adolescence is a pivotal time in a girl's life. The development of educational, physical, psychosocial, familial, political and economic capabilities enable girls to reach their full potential and contribute to the wellbeing of their families and society. However, progress is still significantly constrained by discriminatory gender norms and the related attitudes and practices which restrict girls’ horizons, restrain their ambition and, if unfettered, allow exploitation and abuse. Empowering Adolescent Girls in Developing Countries explores the detrimental impact of discriminatory gender norms on adolescent girls’ lives across very different contexts. Grounded in four years of in-depth research in Ethiopia, Nepal, Uganda and Viet Nam, the book adopts a holistic approach, recognising the inter-related nature of capabilities and the importance of local context. By exploring the theory of gendered norm change, contextualising and examining socialisation processes, the book identifies the patriarchal vested interests in power, authority and moral privilege, which combine in attempts to restrict and control girls’ lives. Throughout the book, Empowering Adolescent Girls in Developing Countries demonstrates how efforts to develop more egalitarian gender norms can enable disadvantaged adolescent girls to change the course of their lives and contribute to societal change. Accessible and informative, the book is perfect for policy makers, think tanks, NGOs, activists, academics and students of gender and development studies."}},"datafield":{"ind1":"8","ind2":" ","tag":"024","subfield":{"code":"a","value":"https://library.oapen.org/handle/20.500.12657/101821"}},"datafield":{"ind1":"0","ind2":"0","tag":"245","subfield":{"code":"a","value":"Empowering Adolescent Girls in Developing Countries"}}}
,"":{"header":{"identifier":{"value":"oai:library.oapen.org:20.500.12657/101541"},"datestamp":{"value":"2025-05-15T07:12:33Z"},"setSpec":{"value":"com_20.500.12657_5"},"setSpec":{"value":"col_20.500.12657_6"}},"metadata":{"record":{"xsi:schemaLocation":"http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd","leader":{"value":"00925njm 22002777a 4500"},"datafield":{"ind1":" ","ind2":" ","tag":"042","subfield":{"code":"a","value":"dc"}},"datafield":{"ind1":" ","ind2":" ","tag":"260","subfield":{"code":"c","value":"2021"}},"datafield":{"ind1":" ","ind2":" ","tag":"520","subfield":{"code":"a","value":"This book is about the invisible or subtle nature of food and farming, and also about the nature of existence. Everything that we know (and do not know) about the physical world has a subtle counterpart which has been scarcely considered in modernist farming practice and research. If you think this book isn’t for you, if it appears more important to attend to the pressing physical challenges the world is facing before having the luxury of turning to such subtleties, then think again. For it could be precisely this worldview – the one prioritises the physical-material dimension of reality - that helped get us into this situation in the first place. Perhaps we need a different worldview to get us out? This book makes a foundational contribution to the discipline of Subtle Agroecologies, a nexus of indigenous epistemologies, multidisciplinary advances in wave-based and ethereal studies, and the science of sustainable agriculture. Not a farming system in itself, Subtle Agroecologies superimposes a non-material dimension upon existing, materially-based agroecological farming systems. Bringing together 43 authors from 12 countries and five continents, from the natural and social sciences as well as the arts and humanities, this multi-contributed book introduces the discipline, explaining its relevance and potential contribution to the field of Agroecology. Research into Subtle Agroecologies may be described as the systematic study of the nature of the invisible world as it relates to the practice of agriculture, and to do this through adapting and innovating with research methods, in particular with those of a more embodied nature, with the overall purpose of bringing and maintaining balance and harmony. Such research is an open-minded inquiry, its grounding being the lived experiences of humans working on, and with, the land over several thousand years to the present. By reclaiming and reinterpreting the perennial relationship between humans and nature, the implications would revolutionise agriculture, heralding a new wave of more sustainable farming techniques, changing our whole relationship with nature to one of real collaboration rather than control, and ultimately transforming ourselves."}},"datafield":{"ind1":"8","ind2":" ","tag":"024","subfield":{"code":"a","value":"https://library.oapen.org/handle/20.500.12657/101541"}},"datafield":{"ind1":"0","ind2":"0","tag":"245","subfield":{"code":"a","value":"Subtle Agroecologies"}}}

pretty printed json:

{
  "header" : {
    "status" : "deleted",
    "identifier" : {
      "value" : "oai:library.oapen.org:20.500.12657/24255"
    },
    "datestamp" : {
      "value" : "2025-05-15T11:23:58Z"
    },
    "setSpec" : {
      "value" : "com_20.500.12657_5"
    },
    "setSpec" : {
      "value" : "col_20.500.12657_6"
    }
  }
}
{
  "header" : {
    "identifier" : {
      "value" : "oai:library.oapen.org:20.500.12657/101622"
    },
    "datestamp" : {
      "value" : "2025-05-15T07:12:23Z"
    },
    "setSpec" : {
      "value" : "com_20.500.12657_5"
    },
    "setSpec" : {
      "value" : "col_20.500.12657_6"
    }
  },
  "metadata" : {
    "record" : {
      "schemaLocation" : "http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd",
      "leader" : {
        "value" : "00925njm 22002777a 4500"
      },
      "datafield" : {
        "ind1" : " ",
        "ind2" : " ",
        "tag" : "042",
        "subfield" : {
          "code" : "a",
          "value" : "dc"
        }
      },
      "datafield" : {
        "ind1" : " ",
        "ind2" : " ",
        "tag" : "720",
        "subfield" : {
          "code" : "a",
          "value" : "Bladini, Moa"
        },
        "subfield" : {
          "code" : "e",
          "value" : "author"
        }
      },
      "datafield" : {
        "ind1" : " ",
        "ind2" : " ",
        "tag" : "720",
        "subfield" : {
          "code" : "a",
          "value" : "Svensson, Eva-Maria"
        },
        "subfield" : {
          "code" : "e",
          "value" : "author"
        }
      },
      "datafield" : {
        "ind1" : " ",
        "ind2" : " ",
        "tag" : "260",
        "subfield" : {
          "code" : "c",
          "value" : "2025"
        }
      },
      "datafield" : {
        "ind1" : " ",
        "ind2" : " ",
        "tag" : "520",
        "subfield" : {
          "code" : "a",
          "value" : "Providing an innovative approach to understanding and addressing anti-gender hate speech, this book focuses on its effects on everyday life, how it is handled within legal frameworks, and how it challenges democratic principles. Unlike previous research, which is often limited to currently regulated forms of hate speech, this book is focused on gender-based hate speech, demonstrating its broader social and ideological impacts. By contextualising freedom of expression within democratic values, it offers a new framework for addressing anti-gender hate speech as a threat to both individual dignity and societal diversity. Through this lens, the authors highlight how anti-gender hate speech represents not only a breach of individual rights but also a structural challenge to democracy, requiring responses that reflect these dual dimensions. The book provides a unique contribution to both legal scholarship and democratic theory by rethinking the balance between gender equality and freedom of expression as cooperative as well as inherently contextual and inclusive. Rethinking Online Anti-Gender Hate Speech will be of great interest to students and scholars of Criminology, Law, Media and Communication, Sociology, and Women’s and Gender Studies."
        }
      },
      "datafield" : {
        "ind1" : "8",
        "ind2" : " ",
        "tag" : "024",
        "subfield" : {
          "code" : "a",
          "value" : "https://library.oapen.org/handle/20.500.12657/101622"
        }
      },
      "datafield" : {
        "ind1" : "0",
        "ind2" : "0",
        "tag" : "245",
        "subfield" : {
          "code" : "a",
          "value" : "Rethinking Online Anti-Gender Hate Speech"
        }
      }
    }
,
    "" : {
      "header" : {
        "identifier" : {
          "value" : "oai:library.oapen.org:20.500.12657/101821"
        },
        "datestamp" : {
          "value" : "2025-05-16T01:26:09Z"
        },
        "setSpec" : {
          "value" : "com_20.500.12657_5"
        },
        "setSpec" : {
          "value" : "col_20.500.12657_6"
        }
      },
      "metadata" : {
        "record" : {
          "schemaLocation" : "http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd",
          "leader" : {
            "value" : "00925njm 22002777a 4500"
          },
          "datafield" : {
            "ind1" : " ",
            "ind2" : " ",
            "tag" : "042",
            "subfield" : {
              "code" : "a",
              "value" : "dc"
            }
          },
          "datafield" : {
            "ind1" : " ",
            "ind2" : " ",
            "tag" : "260",
            "subfield" : {
              "code" : "c",
              "value" : "2018"
            }
          },
          "datafield" : {
            "ind1" : " ",
            "ind2" : " ",
            "tag" : "520",
            "subfield" : {
              "code" : "a",
              "value" : "The Open Access version of this book, available at http://www.tandfebooks.com/doi/view/10.4324/9781315180250, has been made available under a Creative Commons Attribution-Non Commercial-No Derivatives 4.0 license Adolescence is a pivotal time in a girl's life. The development of educational, physical, psychosocial, familial, political and economic capabilities enable girls to reach their full potential and contribute to the wellbeing of their families and society. However, progress is still significantly constrained by discriminatory gender norms and the related attitudes and practices which restrict girls’ horizons, restrain their ambition and, if unfettered, allow exploitation and abuse. Empowering Adolescent Girls in Developing Countries explores the detrimental impact of discriminatory gender norms on adolescent girls’ lives across very different contexts. Grounded in four years of in-depth research in Ethiopia, Nepal, Uganda and Viet Nam, the book adopts a holistic approach, recognising the inter-related nature of capabilities and the importance of local context. By exploring the theory of gendered norm change, contextualising and examining socialisation processes, the book identifies the patriarchal vested interests in power, authority and moral privilege, which combine in attempts to restrict and control girls’ lives. Throughout the book, Empowering Adolescent Girls in Developing Countries demonstrates how efforts to develop more egalitarian gender norms can enable disadvantaged adolescent girls to change the course of their lives and contribute to societal change. Accessible and informative, the book is perfect for policy makers, think tanks, NGOs, activists, academics and students of gender and development studies."
            }
          },
          "datafield" : {
            "ind1" : "8",
            "ind2" : " ",
            "tag" : "024",
            "subfield" : {
              "code" : "a",
              "value" : "https://library.oapen.org/handle/20.500.12657/101821"
            }
          },
          "datafield" : {
            "ind1" : "0",
            "ind2" : "0",
            "tag" : "245",
            "subfield" : {
              "code" : "a",
              "value" : "Empowering Adolescent Girls in Developing Countries"
            }
          }
        }
,
        "" : {
          "header" : {
            "identifier" : {
              "value" : "oai:library.oapen.org:20.500.12657/101541"
            },
            "datestamp" : {
              "value" : "2025-05-15T07:12:33Z"
            },
            "setSpec" : {
              "value" : "com_20.500.12657_5"
            },
            "setSpec" : {
              "value" : "col_20.500.12657_6"
            }
          },
          "metadata" : {
            "record" : {
              "schemaLocation" : "http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd",
              "leader" : {
                "value" : "00925njm 22002777a 4500"
              },
              "datafield" : {
                "ind1" : " ",
                "ind2" : " ",
                "tag" : "042",
                "subfield" : {
                  "code" : "a",
                  "value" : "dc"
                }
              },
              "datafield" : {
                "ind1" : " ",
                "ind2" : " ",
                "tag" : "260",
                "subfield" : {
                  "code" : "c",
                  "value" : "2021"
                }
              },
              "datafield" : {
                "ind1" : " ",
                "ind2" : " ",
                "tag" : "520",
                "subfield" : {
                  "code" : "a",
                  "value" : "This book is about the invisible or subtle nature of food and farming, and also about the nature of existence. Everything that we know (and do not know) about the physical world has a subtle counterpart which has been scarcely considered in modernist farming practice and research. If you think this book isn’t for you, if it appears more important to attend to the pressing physical challenges the world is facing before having the luxury of turning to such subtleties, then think again. For it could be precisely this worldview – the one prioritises the physical-material dimension of reality - that helped get us into this situation in the first place. Perhaps we need a different worldview to get us out? This book makes a foundational contribution to the discipline of Subtle Agroecologies, a nexus of indigenous epistemologies, multidisciplinary advances in wave-based and ethereal studies, and the science of sustainable agriculture. Not a farming system in itself, Subtle Agroecologies superimposes a non-material dimension upon existing, materially-based agroecological farming systems. Bringing together 43 authors from 12 countries and five continents, from the natural and social sciences as well as the arts and humanities, this multi-contributed book introduces the discipline, explaining its relevance and potential contribution to the field of Agroecology. Research into Subtle Agroecologies may be described as the systematic study of the nature of the invisible world as it relates to the practice of agriculture, and to do this through adapting and innovating with research methods, in particular with those of a more embodied nature, with the overall purpose of bringing and maintaining balance and harmony. Such research is an open-minded inquiry, its grounding being the lived experiences of humans working on, and with, the land over several thousand years to the present. By reclaiming and reinterpreting the perennial relationship between humans and nature, the implications would revolutionise agriculture, heralding a new wave of more sustainable farming techniques, changing our whole relationship with nature to one of real collaboration rather than control, and ultimately transforming ourselves."
                }
              },