Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

archival missions sidenavsub #2904

Closed
wants to merge 17 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions __tests__/circulars.ts
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ describe('parseEventFromSubject', () => {
const ztfEvent = 'ZTF23aabmzlp'
const epEvent = 'EP241119a'
const frbEvent = 'FRB 20250206A'
const svomEvent = 'sb25021804'

test('handles nonsense subject cases', () => {
expect(parseEventFromSubject('zawxdrcftvgbhnjm')).toBe(undefined)
Expand Down Expand Up @@ -623,6 +624,56 @@ describe('parseEventFromSubject', () => {
expect(parseEventFromSubject(frbSubjectWithHyphen)).toBe(frbEvent)
})
})

describe('SVOM', () => {
test('handles SVOM event names', () => {
const svomSubject =
'SVOM/sb25021804: SVOM detection of a long X-ray transient'
expect(parseEventFromSubject(svomSubject)).toBe(svomEvent)
})

test('handles SVOM event names in misc positions', () => {
const svomSubjectWithNoSpace =
'SVOM detection: SVOM/sb25021804 a long X-ray transient'
expect(parseEventFromSubject(svomSubjectWithNoSpace)).toBe(svomEvent)
})

test('handles SVOM event names with space', () => {
const svomSubjectWithSpace =
'SVOM/sb 25021804: SVOM detection of a long X-ray transient'
expect(parseEventFromSubject(svomSubjectWithSpace)).toBe(svomEvent)
})

test('handles SVOM event names with spaces in misc positions', () => {
const svomSubjectWithSpace =
'SVOM detection: SVOM/sb25021804 a long X-ray transient'
expect(parseEventFromSubject(svomSubjectWithSpace)).toBe(svomEvent)
})

test('handles SVOM event name with an underscore', () => {
const svomSubjectWithUnderscore =
'SVOM/sb_25021804: SVOM detection of a long X-ray transient'
expect(parseEventFromSubject(svomSubjectWithUnderscore)).toBe(svomEvent)
})

test('handles SVOM event names with an underscore in misc positions', () => {
const svomSubjectWithUnderscore =
'SVOM detection: SVOM/sb25021804 a long X-ray transient'
expect(parseEventFromSubject(svomSubjectWithUnderscore)).toBe(svomEvent)
})

test('handles SVOM event name with a hyphen', () => {
const svomSubjectWithHyphen =
'SVOM/sb-25021804: SVOM detection of a long X-ray transient'
expect(parseEventFromSubject(svomSubjectWithHyphen)).toBe(svomEvent)
})

test('handles SVOM event name with a hyphen in misc positions', () => {
const svomSubjectWithHyphen =
'SVOM detection: SVOM/sb-25021804 a long X-ray transient'
expect(parseEventFromSubject(svomSubjectWithHyphen)).toBe(svomEvent)
})
})
})

describe('emailIsAutoReply', () => {
Expand Down
16 changes: 14 additions & 2 deletions app/components/NoticeTypeCheckboxes/NoticeTypeCheckboxes.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
return `${isUpperBound ? '< ' : ''}${humanizedCount(Math.round(rate), singular, plural)} per ${unit}`
}

const NoticeTypes = {
const NoticeTypes: { [key: string]: string[] } = {

Check warning on line 47 in app/components/NoticeTypeCheckboxes/NoticeTypeCheckboxes.tsx

View check run for this annotation

Codecov / codecov/patch

app/components/NoticeTypeCheckboxes/NoticeTypeCheckboxes.tsx#L47

Added line #L47 was not covered by tests
AGILE: [
'AGILE_GRB_GROUND',
'AGILE_GRB_POS_TEST',
Expand Down Expand Up @@ -231,6 +231,18 @@
JsonNoticeTypeLinks.SVOM = '/missions/svom'
}

const displayNoticeTypes: { [key: string]: string[] } = {

Check warning on line 234 in app/components/NoticeTypeCheckboxes/NoticeTypeCheckboxes.tsx

View check run for this annotation

Codecov / codecov/patch

app/components/NoticeTypeCheckboxes/NoticeTypeCheckboxes.tsx#L234

Added line #L234 was not covered by tests
...NoticeTypes,
}

if (selectedFormat == 'voevent') {
displayNoticeTypes['SVOM'] = [

Check warning on line 239 in app/components/NoticeTypeCheckboxes/NoticeTypeCheckboxes.tsx

View check run for this annotation

Codecov / codecov/patch

app/components/NoticeTypeCheckboxes/NoticeTypeCheckboxes.tsx#L239

Added line #L239 was not covered by tests
'gcn.notices.svom.voevent.grm',
'gcn.notices.svom.voevent.eclairs',
'gcn.notices.svom.voevent.mxt',
]
}

if (useFeature('FERMI_GBM_QUICKSTART')) {
JsonNoticeTypes.Fermi = ['gcn.notices.fermi.gbm']
JsonNoticeTypeLinks.Fermi = '/missions/fermi'
Expand Down Expand Up @@ -285,7 +297,7 @@
<NestedCheckboxes
key={selectedFormat}
nodes={Object.entries(
selectedFormat == 'json' ? JsonNoticeTypes : NoticeTypes
selectedFormat == 'json' ? JsonNoticeTypes : displayNoticeTypes
).map(([mission, noticeTypes]) => ({
id: mission,
label: mission,
Expand Down
2 changes: 1 addition & 1 deletion app/lib/utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ export function formatAndNoticeTypeToTopic(
noticeFormat: string,
noticeType: string
) {
return noticeFormat == 'json'
return noticeFormat == 'json' || noticeType.startsWith('gcn.notices')
? noticeType
: `gcn.classic.${noticeFormat}.${noticeType}`
}
Expand Down
4 changes: 2 additions & 2 deletions app/root.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -247,10 +247,10 @@ export function Layout({ children }: { children?: ReactNode }) {
<DevBanner />
<Header />
<NewsBanner>
GCN at AAS 245, Legacy Circulars Address Retirement. See{' '}
New SVOM Notices, Planned Retirement of VOEvent Brokers. See{' '}
<Link
className="usa-link"
to="/news#--gcn-at-aas-245-legacy-circulars-address-retirement"
to="/news#new-svom-notices-retirement-of-voevent-brokers-"
>
news and announcements
</Link>
Expand Down
1 change: 1 addition & 0 deletions app/routes/circulars/circulars.lib.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ const subjectMatchers: SubjectMatcher[] = [
],
[/EP[.\s_-]*(\d{6}[a-z])/i, ([, id]) => `EP${id}`],
[/FRB[.\s_-]*(\d{8}[a-z])/i, ([, id]) => `FRB ${id}`.toUpperCase()],
[/sb[.\s_-]*(\d{8})/i, ([, id]) => `sb${id}`],
]

/** Format a Circular as plain text. */
Expand Down
114 changes: 113 additions & 1 deletion app/routes/docs.client.samples.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,10 @@ and some samples from the FAQs section of the [gcn-kafka-python](https://github.
To contribute your own ideas, make a GitHub pull request to add it to [the Markdown source for this document](https://github.com/nasa-gcn/gcn.nasa.gov/blob/CodeSamples/app/routes/docs.client.samples.md), or [contact us](/contact).

- [Working with Kafka messages](#parsing)

- [HEALPix Sky Maps](#healpix-sky-maps)

## Parsing
## Parsing XML

Within your consumer loop, use the following functions to convert the
content of `message.value()` into other data types.
Expand Down Expand Up @@ -166,6 +167,117 @@ for message in consumer.consume(end[0].offset - start[0].offset, timeout=1):
print(message.value())
```

## Parsing JSON

GCN Notices are distributed through Kafka topics and, for new missions, are typically provided in JSON format. This guide explains how to programmatically read the JSON schema.

Start by subscribing to a Kafka topic and parsing the JSON data.

```python
from gcn_kafka import Consumer
import json

# Kafka consumer configuration
consumer = Consumer(
client_id='fill me in',
client_secret='fill me in',
config={"message.max.bytes": 204194304},
)

# Subscribe to Kafka topic
consumer.subscribe(['igwn.gwalert'])

# Continuously consume and parse JSON data
for message in consumer.consume(timeout=1):
if message.error():
print(message.error())
continue

# Print the topic and message ID
print(f"topic={message.topic()}, offset={message.offset()}")

# Decode message value
value = message.value()
value_str = value.decode("utf-8")
alert_json = json.loads(value_str)
```

## Decoding Embedded Data

Some GCN notices include HEALPix sky maps encoded in [base64](https://datatracker.ietf.org/doc/html/rfc4648.html), a way of encoding binary data into text.
The following code demonstrates how to extract, decode, and save the HEALPix data as a `.fits` file from a received notice. Python's built-in [`base64`](https://docs.python.org/3/library/base64.html#base64.b64encode) module provides the `b64decode` method to simplify the decoding process.

```python
import base64
from astropy.io import fits

# Extract the base64-encoded skymap
skymap_string = alert_json["event"]["skymap"]

# Decode the Base64 string to bytes
decoded_bytes = base64.b64decode(skymap_string)

# Write bytes to a FITS file
with open("skymap.fits", "wb") as fits_file:
fits_file.write(decoded_bytes)

# Open and inspect the FITS file
with fits.open("skymap.fits") as hdul:
hdul.info()
```

## JSON Schema Example for Embedding a FITS File

If you want to include a FITS file in a Notice, add a property to your schema definition in the following format:

```json
"healpix_file": {
"type": "string",
"contentEncoding": "base64",
"contentMediaType": "image/fits",
"description": "Base 64 encoded content of a FITS file"
}
```

JSON Schema supports embedding non-Unicode media within strings using the `contentMediaType` and `contentEncoding`, which enable the distribution of diverse data types. For further details, refer to [non-JSON data](https://json-schema.org/understanding-json-schema/reference/non_json_data.html).

## Encoding Embedded Data

For producer data production pipelines, the following encoding steps convert an input file to a byte string. This guide demonstrates how to encode a file (e.g., skymap.fits) into a `base64` encoded string and submit it to the GCN Kafka broker.

```python
from gcn_kafka import Producer
import base64
import json

# Kafka Producer Configuration
producer = Producer(client_id='fill me in',
client_secret='fill me in',
config={"message.max.bytes": 204194304},

# Set Kafka Topic and Producer Configuration
TOPIC = "igwn.gwalert"

data = {
"event": {}
}

# Read and encode the FITS file
with open("skymap.fits", "rb") as file:
data["event"]["skymap"] = base64.b64encode(file.read())

# Convert dictionary to JSON
json_data = json.dumps(data).encode("utf-8")

# Publish the message
producer.produce(
TOPIC,
json_data,
)

producer.flush()
```

## HEALPix Sky Maps

[HEALPix](https://healpix.sourceforge.io) (<b>H</b>ierarchical, <b>E</b>qual <b>A</b>rea, and iso-<b>L</b>atitude <b>Pix</b>elisation) is a scheme for indexing positions on the unit sphere.
Expand Down
5 changes: 5 additions & 0 deletions app/routes/missions.agile.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import { redirect } from '@remix-run/react'

Check warning on line 1 in app/routes/missions.agile.tsx

View check run for this annotation

Codecov / codecov/patch

app/routes/missions.agile.tsx#L1

Added line #L1 was not covered by tests

export async function loader() {
return redirect('../archival/agile')

Check warning on line 4 in app/routes/missions.agile.tsx

View check run for this annotation

Codecov / codecov/patch

app/routes/missions.agile.tsx#L3-L4

Added lines #L3 - L4 were not covered by tests
}
8 changes: 8 additions & 0 deletions app/routes/missions.archival._index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
handle:
breadcrumb: Archival Missions
---

# Archival Missions

Archival missions are no longer operating. We maintain their mission pages to provide records of their GCN participation and historical notices. The archival missions in this list were active at the start of the new GCN (circa 2022) or began after the transition to the new GCN. Missions that operated prior to 2022 are archived at [GCN Classic](https://gcn.gsfc.nasa.gov).
14 changes: 14 additions & 0 deletions app/routes/missions.archival.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
/*!
* Copyright © 2023 United States Government as represented by the
* Administrator of the National Aeronautics and Space Administration.
* All Rights Reserved.
*
* SPDX-License-Identifier: Apache-2.0
*/
import type { BreadcrumbHandle } from '~/root/Title'

export { Outlet as default } from '@remix-run/react'

Check warning on line 10 in app/routes/missions.archival.ts

View check run for this annotation

Codecov / codecov/patch

app/routes/missions.archival.ts#L10

Added line #L10 was not covered by tests

export const handle: BreadcrumbHandle = {

Check warning on line 12 in app/routes/missions.archival.ts

View check run for this annotation

Codecov / codecov/patch

app/routes/missions.archival.ts#L12

Added line #L12 was not covered by tests
breadcrumb: 'Archival Missions',
}
5 changes: 5 additions & 0 deletions app/routes/missions.burstcube.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import { redirect } from '@remix-run/react'

Check warning on line 1 in app/routes/missions.burstcube.tsx

View check run for this annotation

Codecov / codecov/patch

app/routes/missions.burstcube.tsx#L1

Added line #L1 was not covered by tests

export async function loader() {
return redirect('../archival/burstcube')

Check warning on line 4 in app/routes/missions.burstcube.tsx

View check run for this annotation

Codecov / codecov/patch

app/routes/missions.burstcube.tsx#L3-L4

Added lines #L3 - L4 were not covered by tests
}
2 changes: 1 addition & 1 deletion app/routes/missions.svom/route.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import logo from './logo_svom_black.svg'

The [Space-based multi-band astronomical Variable Objects Monitor (SVOM)](https://www.svom.eu/en/the-svom-mission/) is a French-Chinese mission, result of a collaboration between the two national space agencies, [China National Space Administration (CNSA)](https://www.cnsa.gov.cn/english/index.html) and [Centre national d'études spatiales (CNES)](https://cnes.fr/en/projects/svom). SVOM is dedicated to the study of the most powerful transient phenomena, with a particular emphasis on gamma-ray bursts (GRBs).

The SVOM spacecraft carries four multi-wavelength instruments: ECLAIRs, Gamma Ray burst Monitor (GRM), Microchannel X-ray Telescope (MXT) and Visual Telescope (VT). ECLAIRs and GRM can detect gamma-ray transient sources in real-time with localization capabilities for ECLAIRs. SVOM can transfer the alerts in near real-time with a typical latency of less than 30s and perform autonomous slew to allow x-ray and optical follow-up of the source with the smaller field of view instruments: MXT and VT.
The SVOM spacecraft carries four multi-wavelength instruments: ECLAIRs, Gamma Ray burst Monitor (GRM), Microchannel X-ray Telescope (MXT) and Visible Telescope (VT). ECLAIRs and GRM can detect gamma-ray transient sources in real-time with localization capabilities for ECLAIRs. SVOM can transfer the alerts in near real-time with a typical latency of less than 30s and perform autonomous slew to allow x-ray and optical follow-up of the source with the smaller field of view instruments: MXT and VT.

<div className="overflow-table">

Expand Down
30 changes: 19 additions & 11 deletions app/routes/missions.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import { NavLink, Outlet } from '@remix-run/react'
import { GridContainer } from '@trussworks/react-uswds'

import { SideNav } from '~/components/SideNav'
import { SideNav, SideNavSub } from '~/components/SideNav'

Check warning on line 11 in app/routes/missions.tsx

View check run for this annotation

Codecov / codecov/patch

app/routes/missions.tsx#L11

Added line #L11 was not covered by tests
import { useFeature } from '~/root'
import type { BreadcrumbHandle } from '~/root/Title'

Expand All @@ -24,12 +24,6 @@
<NavLink key="." to="." end>
Missions, Instruments, and Facilities
</NavLink>,
<NavLink key="agile" to="agile">
AGILE
</NavLink>,
<NavLink key="burstcube" to="burstcube">
BurstCube
</NavLink>,
<NavLink key="calet" to="calet">
CALET
</NavLink>,
Expand Down Expand Up @@ -82,11 +76,25 @@
<NavLink key="sksn" to="sksn">
Super-Kamiokande
</NavLink>,
useFeature('SVOM') && (
<NavLink key="svom" to="svom">
SVOM
<NavLink key="svom" to="svom">
SVOM
</NavLink>,
<>
<NavLink key="archival" to="archival">
Archival
</NavLink>
),
<SideNavSub
base="archival"
items={[
<NavLink key="agile" to="archival/agile">
AGILE
</NavLink>,
<NavLink key="burstcube" to="archival/burstcube">
BurstCube
</NavLink>,
]}
/>
</>,
]}
/>
</div>
Expand Down
31 changes: 31 additions & 0 deletions app/routes/news._index/route.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,37 @@ import { Anchor } from '~/components/Anchor'

## 2025

<CollectionItem
className="maxw-none grid-container"
variantComponent={
<CollectionCalendarDate datetime={'February 20, 2025'} />
}
>
<CollectionHeading headingLevel="h3">
<Anchor>New SVOM Notices, Retirement of VOEvent Brokers </Anchor>
</CollectionHeading>
<CollectionDescription>

#### SVOM VOEvent Notices Available Over Kafka
The SVOM Team and the GCN team are pleased to announce the availability of SVOM notice types via the [new GCN](https://gcn.nasa.gov) in VOEvent format. These notices can be streamed via [Kafka](https://gcn.nasa.gov/quickstart). They are not available via GCN Classic. The SVOM Team plans to add JSON format notices at a later date.

SVOM Mission Summary: The [Space-based multi-band astronomical Variable Objects Monitor (SVOM)](https://www.svom.eu/en/the-svom-mission/) is a French-Chinese mission, result of a collaboration between the two national space agencies, [China National Space Administration (CNSA)](https://www.cnsa.gov.cn/english/index.html) and [Centre national d'études spatiales (CNES)](https://cnes.fr/en/projects/svom). SVOM mission is dedicated to the study of the most powerful transient phenomena, with a particular emphasis on gamma-ray bursts (GRBs). The SVOM spacecraft carries four multi-wavelength instruments: ECLAIRs, Gamma Ray burst Monitor (GRM), Microchannel X-ray Telescope (MXT) and Visible Telescope (VT).

Notice topics: new SVOM notices provide GRB detection and follow-up observation details. Each of the four instruments has its own dedicated topic to stream notices with their related information, see table below:
- `gcn.notices.svom.voevent.grm`
- `gcn.notices.svom.voevent.eclairs`
- `gcn.notices.svom.voevent.mxt`
- `gcn.notices.svom.voevent.vt`

A more extensive description of the information provided in these new notices is available at https://gcn.nasa.gov/missions/svom.

#### Planned Retirement of GCN Classic VOEvent Brokers
The result of the November 2024 user survey of those utilizing the GCN VOEvent Brokers to receive VOEvent-format Notices with the VOEvent Transport Protocol is overwhelmingly in favor of retiring the brokers and users migrating to receive VOEvent Notices over Kafka. This service is already available for all GCN Classic Notice types with self-subscription via the [Start Streaming Notices Guide](/quickstart). The [GCN Classic VOEvent brokers](https://gcn.gsfc.nasa.gov/voevent.html) will be retired shortly after the conclusion of the O4 gravitational wave network observing run, [currently scheduled for October 7, 2025](https://observing.docs.ligo.org/plan/). We recommend the following actions for users currently utilizing VOEvent via GCN Classic.
- Sign up to receive VOEvent format Notices via [Kafka or email](/docs/notices/consuming)
- [Let the GCN Team know](/contact) when you're ready to unsubscribe from receiving them via the GCN Classic brokers. All GCN Classic VOEvent subscribers will be unsubscribed after the end of O4 and the brokers will be taken offline.
</CollectionDescription>
</CollectionItem>

<CollectionItem
className="maxw-none grid-container"
variantComponent={
Expand Down
Loading