MISP was originally conceived to share IoCs with other researchers. STIX 2 is a language and serialization format used to exchange cyber threat intelligence (CTI). STIX 1 (2012) is nearly as old as MISP (2011), but has been deprecated in favor over STIX 2.

Conversion between the two formats is often necessary, as one might want to ingest MISP data into a TAXII server, or have a MISP server that needs STIX data from a TAXII server. MISP uses its own data format for events and there is no complete overlap between STIX and MISP formats. As a consequence, conversion is lossy, information is lost.

I recently wanted to set up a pipeline to ingest data from a TAXII server into MISP. It is very straightforward if you do not want to modify your data before uploading it to MISP. However, if you need to enrich your MISP data before the upload, for example by adding tags or setting the threat level, things get interesting. Since I ran into some oddities and did not find much help, I’m writing down the steps needed to ingest STIX data into MISP with the intermediary step to enrich the converted data.

Overview of the Pipeline

Here’s the steps for a successful conversion:

1. Download STIX data from TAXII server
2. Convert data to MISP format
3. Create MISP event object
4. Enrich event data
5. Create event via API in MISP

The code snippets below are incomplete (no credentials and such) and unrefined (no error handling), to focus on the important steps. For the intermediary files you might want to use Python’s tempfile instead or clean them up via a cronjob.

Requirements

To successfully convert STIX data to MISP, you need an additional library. Note that there’s several (unmaintained) converters out there, but I use the official one from the MISP project called misp-stix. Also note that from January 1st 2024 onwards, pyMISP requires at least Python 3.10 to run, even though your MISP installation might still use an older Python version.

Here are the third-party requirements:

1
2
3
4

misp-stix
pymisp
stix2
taxii2-client

Directly Upload STIX Data to MISP

If you do not want to modify the STIX data in any way, the upload is a no-brainer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# add your imports

# setup connection
server = taxii2client.v21.Server(taxii_discovery, user=taxii_user, password=taxii_password)
collection = taxii2client.v21.Collection(f'{taxii_root}/collections/{taxii_collection}', user=taxii_user, password=taxii_pass)
# --------- download stix data, see other code snippet --------- #
# save data
filepath_stix = pathlib.Path("/home/alice/stix_data.json")
MemoryStore(stix_data).save_to_file(filepath_stix)

# upload data directly to MISP with no changes
misp = PyMISP(misp_url, misp_key)
misp_response = misp.upload_stix(filepath_stix, version='2')

Enrich MISP Data Before Upload

The library misp-stix is a CLI tool, but can also be used in a Python script. Oddly enough, the function stix_2_to_misp() can only save a MISP event as a serialized JSON object to disk and not directly create a MISPEvent object. I dug a bit through the code, but did not find any documentation: There seems to be no other way than serializing first, then read in the file again to have a MISPEvent object. If you find any other way, feel free to ping me on Mastodon, as I was not able to find another solution (and also wanted to get the job done).

Another pitfall lies in the arguments of stix_2_to_misp(): The code lists two arguments, output_dir and output_name. However, if you use both, output_dir and output_name, it will save the file in the script’s working directory and not in output_dir. Since the code has no docstrings and not much documentation, I am unsure what the developers intended there. Maybe I missed something, but again, I wanted to get the job done. For the script to find the newly-created file, use only output_name with a complete filepath.

Here’s the sample code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

import pathlib

from misp_stix_converter import misp_stix_converter
from pymisp import PyMISP
import pymisp.tools.stix
from stix2 import MemoryStore
import taxii2client.v21

# setup connection
server = taxii2client.v21.Server(taxii_discovery, user=taxii_user, password=taxii_password)
collection = taxii2client.v21.Collection(f'{taxii_root}/collections/{taxii_collection}', user=taxii_user, password=taxii_pass)
# --------- get stix data --------- #
# added_after_date: start date from which to start sync, sample is 5 days ago
added_after_date = (datetime.now() - timedelta(days=5)).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
query_args = {'added_after': added_after_date}
stix_data = [ioc for envelope in taxii2client.v21.as_pages(collection.get_objects, per_request=500, **query_args) for ioc in envelope.get('objects', [])]

filepath_stix = pathlib.Path("stix_data.json")
MemoryStore(stix_data).save_to_file(filepath_stix)

# convert data and save to file
misp_filepath = pathlib.Path("misp_data.json")
_ = misp_stix_converter.stix_2_to_misp(filepath_stix, output_name=misp_filepath)

# open converted data
with misp_filepath.open("r") as fi:
    misp_event_json = fi.read()

# load data into MISP event
event = pymisp.MISPEvent()
event.from_json(misp_event_json)

# enrich event
event.info = "My lovely event full of IoCs"
event.threat_level_id = 1
event.add_tag("tlp:green")

# upload to MISP
misp = PyMISP(misp_url, misp_key)
misp_response = misp.add_event(event)

Someone gracefully provided the code to retrieve data from the TAXII server and allowed me to include it here to benefit the wider community. Thank you.

Image source: “one big server on a mounain. Another server on another mountain. Mountains connected by a bridge”, Nightcafe/Crystal Clear XL.

Last modified on 2024-02-08