Forums

Data saved multiple time into my table

Hello, i receive +- 20 post request and I am trying to save the data into my table but it is saving it many times. As you can see on the red line it seems to repeat itself, up to 4 times, and it adds "new" data from late POST request ( in blue ).

enter image description here

I also have this, but i'm not sur about how to use them . It looks like there is 4 "metrics collectors" and the data also repeat itself 4 times

enter image description here

And this is how i store my data, i just want to store only once the data i get in each new POST request

@app.route(webhook_url, methods=['POST'])
def index():
try:
    data = request.json
    fields_required = ["key","pair","price","timestamp"]
    if not all(field in data for field in fields_required):
        print("missing fields {0} on {1}".format(set(fields_required) - data.keys(), data))
        return jsonify({
            'message' : "missing fields {0} on {1}".format(set(fields_required) - data.keys(),data)
        })
    else:
        return jsonify({"message":"All fields is fine"})
    try:
        if (data['key'] == "imagineifthiswasmypassword"):
            print("Signal received {0} - {1}".format(data["pair"],data["timestamp"]))
            new_job = Job(key=data['key'], pair=data['pair'], price=data['price'], timestamp=data['timestamp'])
            try:
                db.session.add(new_job)
                db.session.commit()
            except Exception as e:
                print("Error adding job to database:", e)
                db.session.rollback()
            print("Signal mis dans totodb {0} - {1}".format(data["ticker"],data["timestamp"]))
        else:
            print('Pas bon key')
    finally:
        print('--->Finis du setup // finfin '+str(datetime.now()))
except BadRequest as e:
    return 'Bad request: {}'.format(e)
hopedestroyed | 8 posts | Jan. 3, 2023, 6:02 p.m. | permalink

The 4 processes that you see are the workers for you web app. When a request comes in, one of them will handle it. Make sure that you are not getting multiple requests to your web app that are providing the same data, which would cause the issue that you are seeing.

Staff glenn | 9498 posts | PythonAnywhere staff | Jan. 3, 2023, 6:33 p.m. | permalink

I have multiple requests but each request is sending only one data ( 1 id into the table ) .The data is never the same. And also i have 10 workers, not 4, and it seems that each process of the 4 processes are handling all the data

hopedestroyed | 8 posts | Jan. 3, 2023, 6:49 p.m. | permalink

What is a "metrics collector", then? Each worker will only handle one request and a request will not be handled by more than one worker.

Staff glenn | 9498 posts | PythonAnywhere staff | Jan. 3, 2023, 6:59 p.m. | permalink

If each worker is handling one request and that it will not be handled by more than one worker, then i must have made an error somewhere, but i don't see where. Lets say i receive 3 post request, the first one : data1, the second one ; data2, the third one data3. The output in the table is, [data1,data2,data3,data1,data2,data3,data1,data2,data3,data1,data2,data3] And i only want [data1,data2,data3] I didn't create any loop. How should i change my code to get the output[data1,data2,data3]

hopedestroyed | 8 posts | Jan. 4, 2023, 1:06 p.m. | permalink

I'm not sure if I'm reading your code correctly (there seem to be some indentation issues in the pasted gist), but it looks like it's not the code handling the actual db insertion, since the code which is supposed to do that is probably not reachable -- you have two return statements above for both logical branches of the if statement. Are you sure the code is being run at all?

Staff pafk | 2973 posts | PythonAnywhere staff | Jan. 4, 2023, 1:37 p.m. | permalink

Sorry, i will indent correctly. This is the whole code

app = Flask(__name__)

#app.app_context().push()
SQLALCHEMY_DATABASE_URI = 'mysql+mysqlconnector://{usern}:{psw}@{hostname}/{databasename}?charset=utf8&collation=utf8mb4_general_ci'.format(
    usern ="something1",
    psw = "something2",
    hostname = "something3.mysql.pythonanywhere-services.com",
    databasename = "something3$something4",
)

app.config["SQLALCHEMY_DATABASE_URI"] = SQLALCHEMY_DATABASE_URI
app.config["SQLALCHEMY_ENGINE_OPTIONS"] = {"pool_recycle": 299}
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False

db=SQLAlchemy(app)

app.config["DEBUG"] = True

# Define the URL of the webhook
webhook_url = '/'
# Function to save data
def save_data(data):
    new_job = Job(key=data['key'], pair=data['pair'], 
        price=data['price'], timestamp=data['timestamp'])
    try:
        db.session.add(new_job)
        db.session.commit()
        db.session.close()
    except Exception as e:
        print("Error adding job to database:", e)
        db.session.rollback()
#
#
#
@app.route(webhook_url, methods=['POST'])
def index():
    try:
        data = request.json
        fields_required = ["key","pair",
            "timeframe","price","timestamp"]
        if not all(field in data for field in fields_required):
            print("missing fields {0} on {1}".format(
                set(fields_required) - data.keys(), data))
            return jsonify({
                'message' : "missing fields {0} on {1}".format(
                     set(fields_required) - data.keys(),data)
            })
        try:
            if (data['key'] == "something5"):
                print("Signal received {0} - {1}".format(data["pair"],data["timestamp"]))
                save_data(data)
                print("Signal put inside my table {0} - {1}".format(data["pair"],data["timestamp"]))
            else:
                print('Not good key')
        finally:
            print('End of handling webhook '+str(datetime.now()))
    except BadRequest as e:
        return 'Bad request: {}'.format(e)
#
#
#This is my table
class Job(db.Model):
    __tablename__ = "mytable"
    id = db.Column(db.Integer, primary_key=True)
    key = db.Column(db.String(64))
    pair = db.Column(db.String(64))
    timeframe = db.Column(db.String(64))
    price = db.Column(db.String(64))
    timestamp = db.Column(db.DateTime)

That is all the code, and yes we go inside the index() function since it i printring in the server log that is was put in the table... But it is doing it 4 times

hopedestroyed | 8 posts | Jan. 4, 2023, 2:09 p.m. | permalink

I forgot to put this , issue resolved 

SQLALCHEMY_ENGINE_OPTIONS = {"pool_recycle": 299}

# Create a connection to the database
engine = create_engine(SQLALCHEMY_DATABASE_URI ,**SQLALCHEMY_ENGINE_OPTIONS)
Session = sessionmaker(engine)
session = Session()
hopedestroyed | 8 posts | Jan. 4, 2023, 2:55 p.m. | permalink

If index is being called multiple times, then that is because it is receiving multiple requests.You can check your access log and you should be able to see those requests being logged.

Staff glenn | 9498 posts | PythonAnywhere staff | Jan. 5, 2023, 10:50 a.m. | permalink