Issue with running tram.py

[kyozen]

COULD NOT CONNECT TO TAXII SERVERS: 'description'
PLEASE UTILIZE THE OFFLINE CAPABILITY FLAG "-FF" FOR OFFLINE DATABASE BUILDING
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<background_tasks() done, defined at tram.py:21> exception=SystemExit()>
Traceback (most recent call last):
File "tram.py", line 34, in background_tasks
await data_svc.insert_attack_stix_data()

data_svc.py", line 54, in insert_attack_stix_data
"description": i['description'].replace('', '').replace('', '').replace(

python3.7/site-packages/stix2/base.py", line 195, in getitem
return self._inner[key]
KeyError: 'description'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
self.run_forever()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
self._run_once()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 1775, in _run_once
handle._run()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "tram.py", line 40, in background_tasks
sys.exit()
SystemExit

Please help, having issues solving this...

[KadeMorton]

This at face value appears to be the same issue as Issue 59
#59

[arif6008]

I am also having the same issue:

CRITICAL:root:!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
COULD NOT CONNECT TO TAXII SERVERS: 'description'
PLEASE UTILIZE THE OFFLINE CAPABILITY FLAG "-FF" FOR OFFLINE DATABASE BUILDING
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<background_tasks() done, defined at tram.py:23> exception=SystemExit()>
Traceback (most recent call last):
File "tram.py", line 36, in background_tasks
await data_svc.insert_attack_stix_data()
File "C:\Users\Arif\TRAM\tram\service\data_svc.py", line 54, in insert_attack_stix_data
"description": i['description'].replace('', '').replace('', '').replace(
File "C:\Users\Arif\AppData\Roaming\Python\Python37\site-packages\stix2\base.py", line 216, in getitem
return self._inner[key]
KeyError: 'description'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\asyncio\base_events.py", line 571, in run_until_complete
self.run_forever()
File "C:\ProgramData\Anaconda3\lib\asyncio\base_events.py", line 539, in run_forever
self._run_once()
File "C:\ProgramData\Anaconda3\lib\asyncio\base_events.py", line 1775, in _run_once
handle._run()
File "C:\ProgramData\Anaconda3\lib\asyncio\events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "tram.py", line 42, in background_tasks
sys.exit()
SystemExit

How to fix it?? Please help.

[arif6008]

I found the solutions here. It needs to change the code in /service/data_svc.py file as given in the below link: https://github.com/jecarr/tram/commit/060b4f2152cf7692aa5c76851a86b441247e4645

[ghost]

I found the solutions here. It needs to change the code in /service/data_svc.py file as given in the below link: jecarr@060b4f2

that workes for me aswell, but now the analyze content doesn't render iike in this post https://github.com/mitre-attack/tram/issues/67 and loading just one file of my queue takes up to 20 minutes

[flyingcat1111]

I found that error as below, but I changed my data_svc.py as that link
said https://github.com/jecarr/tram/commit/060b4f2152cf7692aa5c76851a86b441247e4645 , but the error didn't solved

INFO:root:Welcome to TRAM
DEBUG:root:Will build model from static file
DEBUG:asyncio:Using selector: SelectSelector
DEBUG:root:[#] 0 Existing items in the DB
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<background_tasks() done, defined at tram.py:21> exception=KeyError('description')>
Traceback (most recent call last):
File "tram.py", line 42, in background_tasks
await data_svc.insert_attack_json_data(json_file)
File "D:\coding\python\tram\service\data_svc.py", line 143, in insert_attack_json_data
'description': item['description'],
KeyError: 'description'
INFO:root:[] Found punkt
INFO:root:[] Found stopwords
INFO:root:server starting: 0.0.0.0:9999

[MarleenSteinhoff]

Hi @jecarr, with the latest commit 5321499 I still have the same issue as mentioned above #60 (comment).
With the code change in data_svc.py as mentioned in jecarr@060b4f2 is was able to get TRAM running .
However, this code change is not included in the latest commit...
Is this still a valid solution or do I break something with this code change?

Many thanks in advance!

[jecarr]

Hey @buyushan - thanks for the feedback. It looks like my fix didn't work for you because you've specified the config to build the database locally (and not online). So my forked repo and commit doesn't hit

... line 143, in insert_attack_json_data

(to quote from your stack trace).

Without any fixes applied, KeyErrors therefore will occur:

• In insert_attack_json_data if you build the db locally
• In insert_attack_stix_data if you leave the config as is and fetch data from the taxii server

@buyushan, given this you should now notice a slight difference between your error stack trace and those that have been pasted earlier in this issue by others (i.e. different culprit 'insert_' methods)

I realise @GoldBigDragon came across the same scenario as you and created #79. You should be able to take their branch as is or apply solely GoldBigDragon/tram@321e156 to fix your issue.

For completeness of my branch, I've also updated my PR to include this fix (jecarr/tram@7d357fd). I would be keen to know the local json file you specified in the config because the config currently points to an enterprise-attack.json file which doesn't exist. Even if I point to attack_dict.json which does exist, I will receive KeyErrors much earlier than the 'description' line when calling insert_attack_json_data().

It would be good to know how you triggered the error so we can make insert_attack_json_data() more resilient in the future.

[jecarr]

Hi @MarleenSteinhoff - apologies if this doesn't answer your question and/or I'm stating things you already know. Feel free to let me know if so:

Is this still a valid solution?

For the given error you are facing (KeyError), this is a valid solution. This is because the code is at a point where it is trying to go through attack data. For a single attack entry in this attack data, it attempts to read the description of that attack. Let's take a look at the error stack trace from the beginning of this issue:

data_svc.py", line 54, in insert_attack_stix_data
"description": i['description'].replace('', '').replace('', '').replace(

i is the current attack being read at this point. It just so happens this attack doesn't have a description. Therefore when i['description'] occurs, Python will throw a KeyError because it can't find 'description' belonging to i (the attack). In Python terms, it can't find a key 'description' in dictionary i. Once this error is thrown, the code has exited so tram will not load (see end of the stack trace):

sys.exit()
SystemExit

You mentioned you applied the code and tram was able to run. Let's see what is actually applied there; to quote from that commit, it is this:

..."description": i['description']. ... if hasattr(i, "description") else 'No description provided', ...

In Python, this is a ternary operator assignment. It's a shorthand for assigning a variable based on a condition. For example, I could have x = 5 if False else 7. Because False is not True, x becomes 7.

Back to the line: "description": i['description'] -> we have a dictionary (see use of { } around this line and its neighbouring attack fields like name, id, etc). In this dictionary, we have a field 'description' which we are assigning a value for (use of colon, :).

As for the fix: this if statement is checking if the attack (i) has a key 'description'. If it does, it can proceed as normal accessing the description via i['description'] but if it doesn't, the part i['description'] never gets executed and the KeyError is avoided.

Because this is a KeyError, there are different approaches to this fix so someone might check 'description' in object i differently.

---

Do I break something with this code change?

No, this enables tram to finish loading the attack data so will not break anything. You may come across errors whilst using tram - refer to other existing issues here - but (unless they involve the description in some way) it will not be the fault of this fix.

Definitions of a 'breaking change' may change from person to person. For this repo, it could be a fix that raises a new error is a breaking change. Other repos may have tests for you to run to check your code is fine (and doesn't break other parts). You can see an example in a different repo here codelucas/newspaper#885 where there are green ticks and red crosses next to commits which informs me of breaking something. Back to tram, there are no tests (at time of writing) to tell me if something breaks therefore I trust in having tram loaded to tell me it's not breaking anything.

---

However, this code change is not included in the latest commit...

Short answer: It's not included because these are changes applied after that commit. You can see the timestamp for 5321499 is earlier than the timestamp for jecarr/tram@060b4f2.

Longer answer: This is to do with the use of a different branch.

What are branches? These are different versions of the code you may have for a repo. You usually have branches when you want to make code changes without affecting a default or different version of the repo. You can see the tram repo's branches here:

(You can click 'View all branches' at the bottom of the list or the '20 branches' hyperlink to the side of the list to see details of each branch)

All repositories have a default branch, tram's repo has a default branch of master (as you can see from the image). When you cloned the repo (git clone https...), git would have selected the default branch for you to use. Git is a VCS (version control system). Assuming you haven't done any git commands other than the initial git clone, if you go into the tram folder on a terminal and type git status, you will be told On branch master. You will also be told you modified a file (to apply the KeyError fix).

To go beyond branches, I'll briefly mention forked repositories. This is where there is an original repository but a user wants to own a copy. For example, if you go here, in the top left below the repo name, it is stated my jecarr/tram repo is forked from mitre-attack/tram. My jecarr/tram repo therefore has a different default branch, main, which you are told This branch is even with mitre-attack:master. There's that commit 5321499!

When I use my default branch of jecarr/tram, I face the KeyError you faced. I want to apply my fix. So I created an attack-keyerror branch. You'll spot that in the list of branches in jecarr/tram but not mitre-attack/tram. Like many repositories on github, the security settings are such that I can't create a branch in a repo that is not mine/I have permission to contribute to, hence my fix is in a forked repo.

Where it says Create branch: ... from 'main', this means the new branch will contain everything it has in branch main for me to build off of (add code, remove code, etc) in the new branch.

With this separate branch attack-keyerror, I have the goal for it to be merged back in the mitre-attack/tram default branch. This is so people who git clone in the future will have that fix. But this isn't my repo; my change needs to be code reviewed by someone who maintains the mitre-attack/tram repo to determine if it can be merged into their default branch. If you scroll up to the top of this issue and see its info on the right, I have linked a pull request there. If you click it or hover over it, you'll see something like mitre-attack:master <- jecarr:attack-keyerror. This reflects that I want to merge attack-keyerror from my jecarr/tram repo into master of the mitre-attack/tram repo.

Now that I have a separate branch, it is down to me to maintain it. Future commits may occur in mitre-attack/tram:master. I'll then need to ensure my forked repo and its attack-keyerror branch includes such commits. As someone who wants to verify this, you could check the latest commit in the default branch of mitre-attack/tram (as you have done, it being 5321499 at time of writing). You can then view my attack-keyerror branch and see if it is behind master (missing commits) or is even with it (as we saw earlier).

Whilst viewing the attack-keyerror branch (left), you are told it is ahead of mitre-attack/tram:master by 7 commits. That means in addition to being even with the master branch, there have been more commits. If you click the '102 commits' hyperlink, you can see the timeline of commits and can see the 5321499 is present.

Therefore my fix is not included in the latest commit because it is from a forked repo. It is also committed later than the latest commit for the mitre-attack/tram master branch and is pending a merge via the linked pull request.

Hope that helps!

[MarkDavidson]

Hello @kyozen and thank you for the bug report. TRAM has moved to https://github.com/center-for-threat-informed-defense/tram and the bug has been fixed in that repository so I am closing this issue. Thank you!